Logs indicate successful update but image is not actually updated (digest strategy, registry is ECR)

[Dwisf]

Describe the bug I setup the application to use the digest strategy. The registry is the AWS ECR. In the log the image updater shows that it identified the image needs updating, and eventually says it has successfully updated the image. However, the image is not actually updated.

Thinking that I have all the setup required, am I still missing something here? Or is this a bug?

The registry setup:

    - name: ECR
      api_url: https://238231219538.dkr.ecr.us-west-2.amazonaws.com
      prefix: ""
      ping: yes
      default: true
      insecure: no
      tagsortmode: none
      credentials: ext:/scripts/ecr-login.sh
      credsexpire: 11h

The annotation in argocd Application:

...
    argocd-image-updater.argoproj.io/image-list: >-
      my-app=xxx.dkr.ecr.us-west-2.amazonaws.com/my-app:2.85.0
    argocd-image-updater.argoproj.io/my-app.pull-secret: 'ext:/scripts/ecr-login.sh'
    argocd-image-updater.argoproj.io/my-app.update-strategy: digest
...

Expected behavior Expected that the image is updated and container is running the updated image.

Version argocd-image-updater: v0.12.0+aee153d BuildDate: 2022-03-14T12:45:27Z GitCommit: https://github.com/argoproj-labs/argocd-image-updater/commit/aee153dabeb8b592e4d091c933ae4f77181db653 GoVersion: go1.17.8 GoCompiler: gc Platform: linux/amd64

Logs Here is the relevant trace log (I have a total 22 images being setup to be tracked with the image updater)

time="2022-05-03T17:54:26Z" level=info msg="argocd-image-updater v0.12.0+aee153d starting [loglevel:TRACE, interval:2m0s, healthport:8080]"
time="2022-05-03T17:54:26Z" level=warning msg="commit message template at /app/config/commit.template does not exist, using default"
time="2022-05-03T17:54:26Z" level=debug msg="Successfully parsed commit message template"
time="2022-05-03T17:54:26Z" level=debug msg="rate limiting is disabled" prefix= registry="https://xxx.dkr.ecr.us-west-2.amazonaws.com"
time="2022-05-03T17:54:26Z" level=debug msg="Setting default registry endpoint to "
time="2022-05-03T17:54:26Z" level=debug msg="Previous default registry was docker.io"
time="2022-05-03T17:54:26Z" level=info msg="Loaded 1 registry configurations from /app/config/registries.conf"
time="2022-05-03T17:54:26Z" level=debug msg="Creating in-cluster Kubernetes client"
time="2022-05-03T17:54:26Z" level=info msg="ArgoCD configuration: [apiKind=kubernetes, server=argocd-server.argocd, auth_token=false, insecure=false, grpc_web=false, plaintext=false]"
time="2022-05-03T17:54:26Z" level=info msg="Starting health probe server TCP port=8080"
time="2022-05-03T17:54:26Z" level=info msg="Starting metrics server on TCP port=8081"
time="2022-05-03T17:54:26Z" level=info msg="Warming up image cache"
time="2022-05-03T17:54:26Z" level=trace msg="processing app 'platformdev-platform' of type 'Helm'" application=platformdev-platform
time="2022-05-03T17:54:26Z" level=debug msg="Processing application platformdev-platform"
...
...
time="2022-05-03T17:57:47Z" level=debug msg="Considering this image for update" alias=my-app application=platformdev-platform image_name=my-app image_tag=2.85.0 registry=xxx.dkr.ecr.us-west-2.amazonaws.com
time="2022-05-03T17:57:47Z" level=debug msg="Using version constraint '2.85.0' when looking for a new tag" alias=my-app application=platformdev-platform image_name=my-app image_tag=2.85.0 registry=xxx.dkr.ecr.us-west-2.amazonaws.com
time="2022-05-03T17:57:47Z" level=trace msg="Found update strategy digest" image_alias=my-app image_digest= image_name=xxx.dkr.ecr.us-west-2.amazonaws.com/my-app image_tag=2.85.0 registry_url=xxx.dkr.ecr.us-west-2.amazonaws.com
time="2022-05-03T17:57:47Z" level=trace msg="No match annotation found" image_alias=my-app image_digest= image_name=xxx.dkr.ecr.us-west-2.amazonaws.com/my-app image_tag=2.85.0 registry_url=xxx.dkr.ecr.us-west-2.amazonaws.com
time="2022-05-03T17:57:47Z" level=trace msg="Using runtime platform constraint linux/amd64" image_alias=my-app image_digest= image_name=xxx.dkr.ecr.us-west-2.amazonaws.com/my-app image_tag=2.85.0 registry_url=xxx.dkr.ecr.us-west-2.amazonaws.com
time="2022-05-03T17:57:47Z" level=trace msg="Fetching credentials for registry https://xxx.dkr.ecr.us-west-2.amazonaws.com"
time="2022-05-03T17:57:47Z" level=info msg=/scripts/ecr-login.sh dir= execID=80e4f
time="2022-05-03T17:57:48Z" level=trace msg="Performing HTTP GET https://xxx.dkr.ecr.us-west-2.amazonaws.com/v2/my-app/tags/list"
time="2022-05-03T17:57:48Z" level=trace msg="Getting manifest for image my-app:2.85.0 (operation 1/1)" alias=my-app application=platformdev-platform image_name=my-app image_tag=2.85.0 registry=xxx.dkr.ecr.us-west-2.amazonaws.com
time="2022-05-03T17:57:48Z" level=trace msg="acquired metadata semaphore" alias=my-app application=platformdev-platform image_name=my-app image_tag=2.85.0 registry=xxx.dkr.ecr.us-west-2.amazonaws.com
time="2022-05-03T17:57:48Z" level=trace msg="Performing HTTP GET https://xxx.dkr.ecr.us-west-2.amazonaws.com/v2/my-app/manifests/2.85.0"
time="2022-05-03T17:57:48Z" level=trace msg="Manifest digest is d3291b4876217dd80af9c8faef4e159258721fcc13d204e76cd2cfded8918ef8" alias=my-app application=platformdev-platform image_name=my-app image_tag=2.85.0 registry=xxx.dkr.ecr.us-west-2.amazonaws.com
time="2022-05-03T17:57:48Z" level=trace msg="v2 SHA digest is sha256:121c00b1c934bbd61bee24e86b819e6ff61c69a6d074ba49b1ab499f320de883" alias=my-app application=platformdev-platform image_name=my-app image_tag=2.85.0 registry=xxx.dkr.ecr.us-west-2.amazonaws.com
time="2022-05-03T17:57:48Z" level=trace msg="Performing HTTP GET https://xxx.dkr.ecr.us-west-2.amazonaws.com/v2/my-app/blobs/sha256:d3291b4876217dd80af9c8faef4e159258721fcc13d204e76cd2cfded8918ef8"
time="2022-05-03T17:57:48Z" level=trace msg="Performing HTTP GET https://prod-us-west-2-starport-layer-bucket.s3.us-west-2.amazonaws.com/933c01-xxx-66bd9637-8888-47c9-4792-8dc10d0aec43/adfef29a-8b12-4f27-8fa2-51d7e4d1a500?X-Amz-Security-Token=zzz"
time="2022-05-03T17:57:48Z" level=trace msg="Found date 1970-01-01 00:00:00 +0000 UTC" alias=my-app application=platformdev-platform image_name=my-app image_tag=2.85.0 registry=xxx.dkr.ecr.us-west-2.amazonaws.com
time="2022-05-03T17:57:48Z" level=trace msg="released semaphore and terminated waitgroup"
time="2022-05-03T17:57:48Z" level=trace msg="List of available tags found: [2.85.0]" alias=my-app application=platformdev-platform image_name=my-app image_tag=2.85.0 registry=xxx.dkr.ecr.us-west-2.amazonaws.com
time="2022-05-03T17:57:48Z" level=trace msg="Finding out whether to consider 2.85.0 for being updateable" image="xxx.dkr.ecr.us-west-2.amazonaws.com/my-app:2.85.0"
time="2022-05-03T17:57:48Z" level=debug msg="found 1 from 1 tags eligible for consideration" image="xxx.dkr.ecr.us-west-2.amazonaws.com/my-app:2.85.0"
time="2022-05-03T17:57:48Z" level=trace msg="Setting dummy digest for image xxx.dkr.ecr.us-west-2.amazonaws.com/my-app:2.85.0"
time="2022-05-03T17:57:48Z" level=info msg="Setting new image to xxx.dkr.ecr.us-west-2.amazonaws.com/my-app@sha256:121c00b1c934bbd61bee24e86b819e6ff61c69a6d074ba49b1ab499f320de883" alias=my-app application=platformdev-platform image_name=my-app image_tag=dummy registry=xxx.dkr.ecr.us-west-2.amazonaws.com
time="2022-05-03T17:57:48Z" level=debug msg="target parameters: image-spec= image-name=image.name, image-tag=image.tag" application=platformdev-platform image=xxx.dkr.ecr.us-west-2.amazonaws.com/my-app
time="2022-05-03T17:57:48Z" level=info msg="Successfully updated image 'xxx.dkr.ecr.us-west-2.amazonaws.com/my-app@dummy' to 'xxx.dkr.ecr.us-west-2.amazonaws.com/my-app@sha256:121c00b1c934bbd61bee24e86b819e6ff61c69a6d074ba49b1ab499f320de883', but pending spec update (dry run=false)" alias=my-app application=platformdev-platform image_name=my-app image_tag=dummy registry=xxx.dkr.ecr.us-west-2.amazonaws.com
time="2022-05-03T17:57:48Z" level=debug msg="Using commit message: "
time="2022-05-03T17:57:48Z" level=info msg="Committing 22 parameter update(s) for application platformdev-platform" application=platformdev-platform
time="2022-05-03T17:57:48Z" level=info msg="Successfully updated the live application spec" application=platformdev-platform
time="2022-05-03T17:57:48Z" level=info msg="Processing results: applications=1 images_considered=22 images_skipped=0 images_updated=22 errors=0"
....

[tamanh123456]

Hi @Dwisf , I think I resolved the problem. Did you use helm chart for the deployment on cluster ?? I fix it with format helm chart values like this

image: 
  repository: myrepository/nginx
  tag: 1.20.0
  pullPolicy: IfNotPresent

[Dwisf]

Still having the issue, I wonder if someone here has encountered similar issue and/or can suggest way to resolve this. Currently I'm looking at an alternative solution not involving the Image Updater. But it would be nice if I can use the Image Updater for this use case.

[TheDukeDK]

@Dwisf - I ran into this issue and finally figured out the annotations to make this work with write-back. See my comment on #186.

Maybe this will help.

[Dwisf]

Thanks @TheDukeDK for sharing your info. In our case, we are using the default write-back method, which is "argocd" (which update the application resource in the cluster). Can you tell what specific annotation(s) that makes the difference in your case?

I updated an image to include more annotations (based on your example), like the following, but this doesn't solve the issue:

    argocd-image-updater.argoproj.io/cdc-listing-kstream.force-update: 'true'
    argocd-image-updater.argoproj.io/cdc-listing-kstream.helm.image-name: xxx.dkr.ecr.us-west-2.amazonaws.com/cdc-listing-kstream
    argocd-image-updater.argoproj.io/cdc-listing-kstream.helm.image-tag: 2.90.0
    argocd-image-updater.argoproj.io/cdc-listing-kstream.pull-secret: 'ext:/scripts/ecr-login.sh'
    argocd-image-updater.argoproj.io/cdc-listing-kstream.update-strategy: digest

Also can you possibly share the log output of the image updater, so I can see if there is any difference in the log? The log that I have is mentioned in this ticket (in the description).

[vampire-yuta]

Hello:smile:
I am also facing the similar probrem. :cry:

ArgoCD hosting for EKS Ver1.22 . ImageRepository is ECR. I think authentication feature is fine. Because logged success messages.

probrem

• Application Event is logged success messages. But Deployment is not start new deploying.

  Normal  ImagesUpdated  6m52s  ArgocdImageUpdater  Successfully updated application 'test'
  Normal  ImagesUpdated  4m52s  ArgocdImageUpdater  Successfully updated application 'test'
  Normal  ImagesUpdated  2m51s  ArgocdImageUpdater  Successfully updated application 'test'
  Normal  ImagesUpdated  51s    ArgocdImageUpdater  Successfully updated application 'test'

• My manifest file is this.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: truevue
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/write-back-method: argocd
    argocd-image-updater.argoproj.io/image-list: web=xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/web:latest
    argocd-image-updater.argoproj.io/web.update-strategy: digest

Please let me know if there is any better way or to try 🙏

[aditara]

Good Morning guys,

couple days ago I've experienced the same issue as mentioned in this forum. I had checked logs in argocd image updater pod, and I found out some messages that argocd has successfully update image in related app. however it turned out was wrong.

and here the things I've done to make it right :

1. Delete all kubernetes resources owned by argocd namespace
2. Delete namespace of argocd
3. Reproduce argocd by creating it using this command : kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml (Note : please make sure you have created the namespace of argocd earlier)
4. Since the default argocd service is "ClusterIP", I decided to expose them using ingress. You can create your own ingress for your argocd accordingly.
5. Edit configmap of "argocd-cm". add these following lines :

data:
  resource.customizations: |
    networking.k8s.io/Ingress:
      health.lua: |
        hs = {}
        hs.status = "Healthy"
        return hs

6. Restart argocd deployment : kubectl rollout restart deploy -n argocd argocd-server
7. Now login to your registry account through CLI (in my case, I am using docker hub registry) : docker login
8. Create secret to store registry credential : kubectl -n argocd create secret generic <secret_name> --from-file=.dockerconfigjson=/root/.docker/config.json --type=kubernetes.io/dockerconfigjson
9. Modify configmap of "argocd-image-updater-config". add these following lines :

data:
  log.level: debug
  registries.conf: |
	registries:
	- name: Docker Hub
	  prefix: docker.io
	  api_url: https://index.docker.io/
	  ping: yes
	  ## this is detail of secret we've created before ##
	  ## format = pullsecret:<namespace>/<secret_name> ##
	  credentials: pullsecret:argocd/dockerhub-johndoe

10. I am not pretty sure whether it's mandatory to do. But I suggest you to do it. kill or purge pod of argocd-image-updater everytime you make changes in its configmap.
11. Access your argocd web UI using its default password and create your own app that has been created in GitHub/ GitLab earlier. (Do not enable the auto sync feature)
12. Modify your app through CLI. Add these lines :

  annotations:
    argocd-image-updater.argoproj.io/git-branch: main
    argocd-image-updater.argoproj.io/image-list: alias=docker.io/<dockerhub-account-name>/<repository-name>
    argocd-image-updater.argoproj.io/alias.force-update: "true"
    argocd-image-updater.argoproj.io/alias.update-strategy: latest
    argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/<Git-access-token-name>

13. Enable auto sync of argocd App through argocd web UI
14. Wait for several minutes to take effect on it.

These are couple things I've done with my argocd within my kubernetes cluster. And I prove it on my own it works. Please do not hesitate if you guys have other questions or suggestions.

Hopefully this brief workaround could help you who experienced the same issue.

[nahum-landa]

registries:
  - name: ECR
    api_url: https://XXX.dkr.ecr.XXXX.amazonaws.com
    prefix: XXX.dkr.ecr.XXXX.amazonaws.com   # before v0.12 this needed to be set empty, prefix: ""
    default: true  # not supported before v0.12
    ping: yes
    insecure: no
    credentials: ext:/scripts/ecr-login.sh  #script name should match here and in authScripts
    credsexpire: 11h
    tagsortmode: none
    
    fixes this.
    notice the tag sort mode

[amohsenter09-github]

Hey everyone !! seems the login.sh scripts expect syntax : formate. I am getting this error while try to authenticate the ECR time="2023-02-01T12:30:33Z" level=error msg="Could not set registry endpoint credentials: invalid script output, must be single line with syntax :" alias= application=myapplication image_name=myapplication image_tag=latest registry=xxxxx.dkr.ecr.ca-central-1.amazonaws.com time="2023-02-01T12:30:33Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=1" can someone assist this our configuration in helm registries.conf: | registries: - api_url: https://xxxx.dkr.ecr.ca-central-1.amazonaws.com credentials: ext:/scripts/ecr-login.sh credsexpire: 10h insecure: false name: ECR ping: true prefix: xxxx.dkr.ecr.ca-central-1.amazonaws.com

[bebosudo]

It took me a while to figure out that I was missing the helm parameter names that instruct ArgoCD on what Helm values to update, see https://argocd-image-updater.readthedocs.io/en/stable/configuration/images/#specifying-helm-parameter-names

    argocd-image-updater.argoproj.io/write-back-method: argocd
    argocd-image-updater.argoproj.io/image-list: backend=repo.example.com/image
    argocd-image-updater.argoproj.io/backend.update-strategy: latest
    argocd-image-updater.argoproj.io/backend.helm.image-name: backend.image.repository
    argocd-image-updater.argoproj.io/backend.helm.image-tag: backend.image.tag

[kyleli666]

If it's related to Image updater + ApplicationSet, then there is a work arround by changing ApplicationSet controller policy. See my comment here https://github.com/argoproj/applicationset/issues/547#issuecomment-1674693159