ArgoCD Image Updater logs says image is updated, but it's not.

[mrksph]

Describe the bug When using ArgoCD with ArgoCD Image Updater (v0.15.0, also tested with v0.14.0) and ApplicationSet to deploy a demo API to a k3s cluster, the Image Updater successfully updates the image in the ArgoCD application spec, but the pod does not restart to pull the updated image. The updated image is hosted on GitHub Container Registry (GHCR) as a private repository, and credentials are correctly configured.

ArgoCD Image Updater logs shows that it correctly downloads and updates the image (I see the image download count increase in the repo packages page) but says it is pending spec update (dry run=false):

Successfully updated image 'ghcr.io/mrksph/demo-api@dummy' to 'ghcr.io/mrksph/demo-api@sha256:3039e1fc83a2e3e2400b842a03fc8b4669a9f9b3ef16466b242d8b489f21637e', but pending spec update (dry run=false)"

I remember I did manage to set this up some months ago. Now that I'm back with it, I can't make it work.

To Reproduce Steps to reproduce the behavior:

1. Deploy a demo API using ArgoCD, ArgoCD Image Updater, and ApplicationSet.
2. Configure the private image from GHCR and ensure credentials are set up correctly.
3. Apply the following folder structure:

   |-argocd-bootstrap.yaml
   |-app-of-apps/
   |--applicationset.yaml
   |-dev/
   |--demo-api/
   |---values.yaml
   |---templates/
   |---# All Helm template files... (deployment, ingress, application, service)
   

4. Files: argocd-bootstrap.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: root-app
namespace: argocd
spec:
project: default
source:
  repoURL: 'https://github.com/mrksph/k3s-manifests'
  targetRevision: HEAD
  path: app-of-apps
  directory:
    recurse: true
destination:
  server: 'https://kubernetes.default.svc'
  namespace: argocd
syncPolicy:
  automated:
    prune: true
    selfHeal: true

applicationset.yaml

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: applications
namespace: argocd
spec:
generators:
  - git:
      repoURL: https://github.com/mrksph/k3s-manifests.git
      revision: HEAD
      directories:
        - path: dev/*     # Match all subdirectories in the dev folder
template:
  metadata:
    name: '{{path.basename}}-{{path[1]}}'
    annotations:
      argocd-image-updater.argoproj.io/image-list: "{{path.basename}}=ghcr.io/<username>/{{path.basename}}:{{path[0]}}"
      argocd-image-updater.argoproj.io/{{path.basename}}.update-strategy: "digest"
  spec:
    project: default
    source:
      repoURL: https://github.com/mrksph/k3s-manifests.git
      targetRevision: HEAD
      path: '{{path}}'
    destination:
      server: https://kubernetes.default.svc
      namespace: '{{path.basename}}-{{path[0]}}'
    syncPolicy:
      automated:
        prune: true
        selfHeal: true
      syncOptions:
        - CreateNamespace=true

dev/demo-api/values.yaml

replicaCount: 1

image:
repository: ghcr.io/<username>/<repository>
pullPolicy: IfNotPresent
tag: <tag> # I built my demo-api with "dev" tag, and set it here

imagePullSecrets:
- name: ghcr-secret

service:
type: ClusterIP
port: 8080

ingress:
# Configure your ingress

resources: {}

nodeSelector: {}

tolerations: []

affinity: {}

5. Observe the logs to see the image update confirmation.
6. Check if the pod restarts to pull the updated image.

Expected behavior After the image is successfully updated in the ArgoCD spec, the associated pod should restart and pull the updated image.

Additional context

• The image repository is private, and credentials have been confirmed as correct.
• Helm charts are used for the deployment templates.
• ArgoCD is configured with ApplicationSet and Image Updater to automate updates.

Version

• ArgoCD Image Updater: v0.15.0 (also tested with v0.14.0)
• ArgoCD: [Version details, if available]
• Kubernetes: [Cluster version, if relevant]

Logs

time="2024-12-06T18:21:29Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2024-12-06T18:21:29Z" level=debug msg="Processing application demo-api-demo-api"
time="2024-12-06T18:21:29Z" level=debug msg="Considering this image for update" alias=demo-api application=demo-api-demo-api image_name=mrksph/demo-api image_tag=dev registry=ghcr.io
time="2024-12-06T18:21:29Z" level=debug msg="Using version constraint 'dev' when looking for a new tag" alias=demo-api application=demo-api-demo-api image_name=mrksph/demo-api image_tag=dev registry=ghcr.io
time="2024-12-06T18:21:30Z" level=debug msg="found 1 from 1 tags eligible for consideration" image="ghcr.io/mrksph/demo-api:dev"
time="2024-12-06T18:21:30Z" level=info msg="Setting new image to ghcr.io/mrksph/demo-api@sha256:3039e1fc83a2e3e2400b842a03fc8b4669a9f9b3ef16466b242d8b489f21637e" alias=demo-api application=demo-api-demo-api image_name=mrksph/demo-api image_tag=dummy registry=ghcr.io
time="2024-12-06T18:21:30Z" level=debug msg="target parameters: image-spec= image-name=image.name, image-tag=image.tag" application=demo-api-demo-api image=ghcr.io/mrksph/demo-api
time="2024-12-06T18:21:30Z" level=info msg="Successfully updated image 'ghcr.io/mrksph/demo-api@dummy' to 'ghcr.io/mrksph/demo-api@sha256:3039e1fc83a2e3e2400b842a03fc8b4669a9f9b3ef16466b242d8b489f21637e', but pending spec update (dry run=false)" alias=demo-api application=demo-api-demo-api image_name=mrksph/demo-api image_tag=dummy registry=ghcr.io
time="2024-12-06T18:21:30Z" level=debug msg="Using commit message: "
time="2024-12-06T18:21:30Z" level=info msg="Committing 1 parameter update(s) for application demo-api-demo-api" application=demo-api-demo-api
time="2024-12-06T18:21:30Z" level=info msg="Successfully updated the live application spec" application=demo-api-demo-api
time="2024-12-06T18:21:30Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=1 errors=0"