Application image is updated but not deployment's image

[cblair]

Describe the bug argocd-image-updater is running, sees valid candidate images from our docker.io repo, and updates them in the k8s Application, but the k8s deployment does not update it's image. Deployment image does have a kustomization.yaml. This is probably just a bone headed configuration issue in argocd-image-updater or in argocd, but I can't find any examples of a full argocd-image-update kustomization implementation.

Updater logs:

time="2023-03-31T17:52:45Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2023-03-31T17:52:45Z" level=debug msg="Processing application doc-storage-ui"
time="2023-03-31T17:52:45Z" level=debug msg="Considering this image for update" alias=doc-storage-ui application=doc-storage-ui image_name=cblairupliv/doc-storage-ui image_tag= registry=
time="2023-03-31T17:52:45Z" level=debug msg="Using no version constraint when looking for a new tag" alias=doc-storage-ui application=doc-storage-ui image_name=cblairupliv/doc-storage-ui image_tag= registry=
time="2023-03-31T17:52:45Z" level=debug msg="could not parse input tag latest as semver: Invalid Semantic Version"
time="2023-03-31T17:52:45Z" level=debug msg="could not parse input tag pr-latest as semver: Invalid Semantic Version"
time="2023-03-31T17:52:45Z" level=debug msg="could not parse input tag staging as semver: Invalid Semantic Version"
time="2023-03-31T17:52:45Z" level=debug msg="found 5 from 5 tags eligible for consideration" image=cblairupliv/doc-storage-ui
time="2023-03-31T17:52:45Z" level=info msg="Setting new image to cblairupliv/doc-storage-ui:0.1.45" alias=doc-storage-ui application=doc-storage-ui image_name=cblairupliv/doc-storage-ui image_tag= registry=
time="2023-03-31T17:52:45Z" level=info msg="Successfully updated image 'cblairupliv/doc-storage-ui' to 'cblairupliv/doc-storage-ui:0.1.45', but pending spec update (dry run=false)" alias=doc-storage-ui application=doc-storage-ui image_name=cblairupliv/doc-storage-ui image_tag= registry=
time="2023-03-31T17:52:45Z" level=debug msg="Using commit message: "
time="2023-03-31T17:52:45Z" level=info msg="Committing 1 parameter update(s) for application doc-storage-ui" application=doc-storage-ui
time="2023-03-31T17:52:45Z" level=info msg="Successfully updated the live application spec" application=doc-storage-ui
time="2023-03-31T17:52:45Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=1 errors=0"

Application updated (notice that Images is the correct latest semver of cblairupliv/doc-storage-ui:0.1.45):

kc describe app -nargocd doc-storage-ui
Name:         doc-storage-ui
Namespace:    argocd
Labels:       <none>
Annotations:  argocd-image-updater.argoproj.io/doc-storage-ui.update-strategy: semver
              argocd-image-updater.argoproj.io/image-list: doc-storage-ui=index.docker.io/cblairupliv/doc-storage-ui
API Version:  argoproj.io/v1alpha1
Kind:         Application
Metadata:
  Creation Timestamp:  2023-03-31T17:14:34Z
  Generation:          18
  Managed Fields:
    API Version:  argoproj.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:argocd-image-updater.argoproj.io/doc-storage-ui.update-strategy:
          f:argocd-image-updater.argoproj.io/image-list:
      ...
    Manager:      argocd-server
    Operation:    Update
    Time:         2023-03-31T17:14:34Z
    API Version:  argoproj.io/v1alpha1
    Fields Type:  FieldsV1
...
    Manager:      argocd-application-controller
    Operation:    Update
    Time:         2023-03-31T17:14:39Z
    API Version:  argoproj.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:source:
          f:kustomize:
            .:
            f:images:
    Manager:         argocd-image-updater
    Operation:       Update
    Time:            2023-03-31T17:14:39Z
  Resource Version:  6218344
  UID:               a777687f-80de-47ac-9f81-30b832476dc3
Spec:
  Destination:
    Namespace:  default
    Server:     https://kubernetes.default.svc
  Project:      default
  Source:
    Kustomize:
      Images:
        cblairupliv/doc-storage-ui:0.1.45
    Path:             argocd/prod/doc-storage-ui/k8s
    Repo URL:         https://github.com/upliv-health/iac
    Target Revision:  main
  Sync Policy:
    Automated:
      Prune:      true
      Self Heal:  true
    Sync Options:
      CreateNamespace=true
Status:
  Health:
    Status:  Healthy

Deployment not updated in k8s:

kc describe deployment doc-storage-ui
Name:                   doc-storage-ui
Namespace:              default
CreationTimestamp:      Fri, 31 Mar 2023 10:34:54 -0700
Labels:                 app=doc-storage-ui
                        app.kubernetes.io/instance=doc-storage-ui
                        tier=web
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=doc-storage-ui,tier=web
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=doc-storage-ui
           tier=web
  Containers:
   doc-storage-ui:
    Image:      index.docker.io/cblairupliv/doc-storage-ui
    Port:       3000/TCP
    Host Port:  0/TCP
    Command:
      node
      server.js
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   doc-storage-ui-68c575f855 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  35m   deployment-controller  Scaled up replica set doc-storage-ui-68c575f855 to 1

Deploymement yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: doc-storage-ui
  labels:
    app: doc-storage-ui
    tier: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: doc-storage-ui
      tier: web
  template:
    metadata:
      labels:
        app: doc-storage-ui
        tier: web
    spec:
      # Credential for private docker.com repository.
      imagePullSecrets:
        - name: regcred
      containers:
      - name: doc-storage-ui
        image: index.docker.io/cblairupliv/doc-storage-ui
        imagePullPolicy: Always
        ports:
          - containerPort: 3000
        command: ["node", "server.js"]

And the kustomize.yaml in the same directory as the deployment.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- cluster-ip.yaml
- deployment.yaml
- loadbalancer.yaml
- service.yaml
images:
- name: doc-storage-ui
  newName: index.docker.io/cblairupliv/doc-storage-ui
  newTag: latest

To Reproduce Steps to reproduce the behavior:

• configure argocd similar to the above in yaml ^
• watch the argocd-image-update pod k8s logs and verify image found and updated
• Verify the Application image is updated as described above
• Verify the deployment image is not updated

Expected behavior The image version to update in the deployment

Additional context Newest installs for argocd and argocd-image-updater as of today.

Version See above

Logs Lots of logs above.

[TreeKat71]

Any update?