argocd-image-updater icon indicating copy to clipboard operation
argocd-image-updater copied to clipboard

Published 20 hours ago • verified publisher icon argoproj-labs

Only second image in list is getting updated

Open joelgriffiths opened this issue 1 year ago • 4 comments

Describe the bug When using argocd image updater with two images, only the second is updated. The first image remains at the initially defined tag version.

It looks like it has something to do with this info message:

time="2024-01-31T20:09:50Z" level=info msg="Successfully updated image 'git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.8' to 'git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.9', but pending spec update (dry run=false)" alias=email application=psa-upon image_name=software-engineers/psa/upon/email-consumer/stg image_tag=v1.0.8 registry="git.internalnet.com:4567"

To Reproduce

I've also tried several variations of this including:

  • Using aliases for pull-secret and strategy
  • Only using aliases for the strategy
  • Stripping aliases out entirely

Images in the registry: sms: v1.0.1 through v1.0.2 email: v1.0.1 through v1.0.9

In this example, the version deployed with sms updates to the most recent image properly, but the email image remains at the originally deployed version (ie sms=1.0.2 and email=1.0.8). When I swap sms and email in the image-list, the results are reversed (sms=1.0.1 and email=1.0.9).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: psa
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd-image-updater.argoproj.io/image-list: sms=git.internalnet.com:4567/software-engineers/psa/sms-consumer/stg:v1.0.x,email=git.internalnet.com:4567/software-engineers/psa/email-consumer/stg:v1.0.x
    argocd-image-updater.argoproj.io/pull-secret: pullsecret:argocd/git-software-engineers-registry-credentials
    argocd-image-updater.argoproj.io/email.update-strategy: semver
    argocd-image-updater.argoproj.io/sms.update-strategy: semver
spec:
  project: default

  source:
    repoURL: [email protected]:netops/kubernetes/k8s-apps.git
    targetRevision: HEAD
    path: stg/psa
  destination:
    server: https://kubernetes.default.svc
    namespace: psa

  syncPolicy:
    syncOptions:
      - CreateNamespace=true

    automated:
      selfHeal: true
      prune: true


kind: Deployment
metadata:
  name: email-consumer
  namespace: psa
  labels:
    app: email-consumer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: email-consumer
  template:
    metadata:
      labels:
        app: email-consumer
    spec:
      containers:
      - name: email-consumer
        image: git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.8
        envFrom:
        - configMapRef:
            name: email-consumer-env-cm
        resources:
          limits:
            memory: "512Mi"
            cpu: "200m"
      imagePullSecrets:
      - name: argocd-deploy-token

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sms-consumer
  namespace: psa
  labels:
    app: sms-consumer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sms-consumer
  template:
    metadata:
      labels:
        app: sms-consumer
    spec:
      containers:
      - name: sms-consumer
        image: git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.1
        envFrom:
        - configMapRef:
            name: sms-consumer-env-cm
        resources:
          limits:
            memory: "512Mi"
            cpu: "200m"
      imagePullSecrets:
      - name: argocd-deploy-token

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ./email-consumer-deployment.yaml
- ./sms-consumer-deployment.yaml
namespace: psa

Steps to reproduce the behavior:

Expected behavior A clear and concise description of what you expected to happen.

Additional context I did manually alter the logs and annotations to remove a customer name from the path and to not expose the name of our internal registry server.

Version Using v0.12.2

Logs Please paste any relevant logs here

Original Configuration - email last
time="2024-01-31T20:03:48Z" level=debug msg="Considering this image for update" alias=email application=psa-upon image_name=software-engineers/psa/upon/email-consumer/stg image_tag=v1.0.9 registry="git.internalnet.com:4567"
time="2024-01-31T20:03:48Z" level=debug msg="Using version constraint 'v1.0.x' when looking for a new tag" alias=email application=psa-upon image_name=software-engineers/psa/upon/email-consumer/stg image_tag=v1.0.9 registry="git.internalnet.com:4567"
time="2024-01-31T20:03:49Z" level=debug msg="found 2 from 2 tags eligible for consideration" image="git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.9"
time="2024-01-31T20:03:49Z" level=debug msg="Image 'git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.9' already on latest allowed version" alias=email application=psa-upon image_name=software-engineers/psa/upon/email-consumer/stg image_tag=v1.0.9 registry="git.internalnet.com:4567"
time="2024-01-31T20:03:49Z" level=debug msg="Considering this image for update" alias=sms application=psa-upon image_name=software-engineers/psa/upon/sms-consumer/stg image_tag=v1.0.1 registry="git.internalnet.com:4567"
time="2024-01-31T20:03:49Z" level=debug msg="Using version constraint 'v1.0.x' when looking for a new tag" alias=sms application=psa-upon image_name=software-engineers/psa/upon/sms-consumer/stg image_tag=v1.0.1 registry="git.internalnet.com:4567"
time="2024-01-31T20:03:49Z" level=debug msg="found 2 from 2 tags eligible for consideration" image="git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.1"
time="2024-01-31T20:03:49Z" level=info msg="Setting new image to git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.2" alias=sms application=psa-upon image_name=software-engineers/psa/upon/sms-consumer/stg image_tag=v1.0.1 registry="git.internalnet.com:4567"
time="2024-01-31T20:03:49Z" level=info msg="Successfully updated image 'git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.1' to 'git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.2', but pending spec update (dry run=false)" alias=sms application=psa-upon image_name=software-engineers/psa/upon/sms-consumer/stg image_tag=v1.0.1 registry="git.internalnet.com:4567"

Swap to sms last
time="2024-01-31T20:09:50Z" level=debug msg="Considering this image for update" alias=sms application=psa-upon image_name=software-engineers/psa/upon/sms-consumer/stg image_tag=v1.0.2 registry="git.internalnet.com:4567"
time="2024-01-31T20:09:50Z" level=debug msg="Using version constraint 'v1.0.x' when looking for a new tag" alias=sms application=psa-upon image_name=software-engineers/psa/upon/sms-consumer/stg image_tag=v1.0.2 registry="git.internalnet.com:4567"
time="2024-01-31T20:09:50Z" level=debug msg="found 2 from 2 tags eligible for consideration" image="git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.2"
time="2024-01-31T20:09:50Z" level=debug msg="Image 'git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.2' already on latest allowed version" alias=sms application=psa-upon image_name=software-engineers/psa/upon/sms-consumer/stg image_tag=v1.0.2 registry="git.internalnet.com:4567"
time="2024-01-31T20:09:50Z" level=debug msg="Considering this image for update" alias=email application=psa-upon image_name=software-engineers/psa/upon/email-consumer/stg image_tag=v1.0.8 registry="git.internalnet.com:4567"
time="2024-01-31T20:09:50Z" level=debug msg="Using version constraint 'v1.0.x' when looking for a new tag" alias=email application=psa-upon image_name=software-engineers/psa/upon/email-consumer/stg image_tag=v1.0.8 registry="git.internalnet.com:4567"
time="2024-01-31T20:09:50Z" level=debug msg="found 2 from 2 tags eligible for consideration" image="git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.8"
time="2024-01-31T20:09:50Z" level=info msg="Setting new image to git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.9" alias=email application=psa-upon image_name=software-engineers/psa/upon/email-consumer/stg image_tag=v1.0.8 registry="git.internalnet.com:4567"
time="2024-01-31T20:09:50Z" level=info msg="Successfully updated image 'git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.8' to 'git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.9', but pending spec update (dry run=false)" alias=email application=psa-upon image_name=software-engineers/psa/upon/email-consumer/stg image_tag=v1.0.8 registry="git.internalnet.com:4567"

joelgriffiths avatar Feb 01 '24 00:02 joelgriffiths

Does anybody have a workaround for this?

joelgriffiths avatar Feb 08 '24 18:02 joelgriffiths

I have some more details on this.

First, the Application resource is not getting updated.

Also, I turned on trace for the logs. Slightly modified to remove private information. In this log, I updated two images in the registry. This is the first run after doing that.

time="2024-02-08T21:29:49Z" level=debug msg="Processing application psa"
time="2024-02-08T21:29:49Z" level=debug msg="Considering this image for update" alias=sms application=psa image_name=software-engineers/psa/upon/sms-consumer/stg image_tag=v1.0.3 registry="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=debug msg="Using version constraint '1.0.x' when looking for a new tag" alias=sms application=psa image_name=software-engineers/psa/upon/sms-consumer/stg image_tag=v1.0.3 registry="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="Found update strategy semver" image_alias=sms image_digest= image_name="git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg" image_tag=1.0.x registry_url="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="No ignore-tags annotation found" image_alias=sms image_digest= image_name="git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg" image_tag=1.0.x registry_url="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="Using runtime platform constraint linux/amd64" image_alias=sms image_digest= image_name="git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg" image_tag=1.0.x registry_url="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="Fetching credentials for registry https://git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="Performing HTTP GET https://git.internalnet.com:4567/v2/software-engineers/ameren/ameren-public-api/stg/tags/list"
time="2024-02-08T21:29:49Z" level=trace msg="Performing HTTP GET https://git.internalnet.com:4567/v2/software-engineers/psa/upon/sms-consumer/stg/tags/list"
time="2024-02-08T21:29:49Z" level=error msg="Could not get tags from registry: name unknown: repository name not known to registry" alias= application=ameren-public-api image_name=software-engineers/ameren/ameren-public-api/stg image_tag=latest registry="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="List of available tags found: [v1.0.3 v1.0.4 v1.0.1 v1.0.2]" alias=sms application=psa image_name=software-engineers/psa/upon/sms-consumer/stg image_tag=v1.0.3 registry="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="Finding out whether to consider v1.0.1 for being updateable" image="git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.3"
time="2024-02-08T21:29:49Z" level=trace msg="Finding out whether to consider v1.0.2 for being updateable" image="git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.3"
time="2024-02-08T21:29:49Z" level=trace msg="Finding out whether to consider v1.0.3 for being updateable" image="git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.3"
time="2024-02-08T21:29:49Z" level=trace msg="Finding out whether to consider v1.0.4 for being updateable" image="git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.3"
time="2024-02-08T21:29:49Z" level=debug msg="found 4 from 4 tags eligible for consideration" image="git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.3"
time="2024-02-08T21:29:49Z" level=info msg="Setting new image to git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.4" alias=sms application=psa image_name=software-engineers/psa/upon/sms-consumer/stg image_tag=v1.0.3 registry="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="Setting Kustomize parameter git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.4" application=psa
time="2024-02-08T21:29:49Z" level=info msg="Successfully updated image 'git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.3' to 'git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg:v1.0.4', but pending spec update (dry run=false)" alias=sms application=psa image_name=software-engineers/psa/upon/sms-consumer/stg image_tag=v1.0.3 registry="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=debug msg="Considering this image for update" alias=email application=psa image_name=software-engineers/psa/upon/email-consumer/stg image_tag=v1.0.11 registry="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=debug msg="Using version constraint '1.0.x' when looking for a new tag" alias=email application=psa image_name=software-engineers/psa/upon/email-consumer/stg image_tag=v1.0.11 registry="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="Found update strategy semver" image_alias=email image_digest= image_name="git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg" image_tag=1.0.x registry_url="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="No ignore-tags annotation found" image_alias=email image_digest= image_name="git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg" image_tag=1.0.x registry_url="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="Using runtime platform constraint linux/amd64" image_alias=email image_digest= image_name="git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg" image_tag=1.0.x registry_url="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="Fetching credentials for registry https://git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="Performing HTTP GET https://git.internalnet.com:4567/v2/software-engineers/psa/upon/email-consumer/stg/tags/list"
time="2024-02-08T21:29:49Z" level=trace msg="List of available tags found: [v1.0.10 v1.0.11 v1.0.12 v1.0.8 v1.0.9]" alias=email application=psa image_name=software-engineers/psa/upon/email-consumer/stg image_tag=v1.0.11 registry="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="Finding out whether to consider v1.0.8 for being updateable" image="git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.11"
time="2024-02-08T21:29:49Z" level=trace msg="Finding out whether to consider v1.0.9 for being updateable" image="git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.11"
time="2024-02-08T21:29:49Z" level=trace msg="Finding out whether to consider v1.0.10 for being updateable" image="git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.11"
time="2024-02-08T21:29:49Z" level=trace msg="Finding out whether to consider v1.0.11 for being updateable" image="git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.11"
time="2024-02-08T21:29:49Z" level=trace msg="Finding out whether to consider v1.0.12 for being updateable" image="git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.11"
time="2024-02-08T21:29:49Z" level=debug msg="found 5 from 5 tags eligible for consideration" image="git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.11"
time="2024-02-08T21:29:49Z" level=info msg="Setting new image to git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.12" alias=email application=psa image_name=software-engineers/psa/upon/email-consumer/stg image_tag=v1.0.11 registry="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=trace msg="Setting Kustomize parameter git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.12" application=psa
time="2024-02-08T21:29:49Z" level=info msg="Successfully updated image 'git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.11' to 'git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.12', but pending spec update (dry run=false)" alias=email application=psa image_name=software-engineers/psa/upon/email-consumer/stg image_tag=v1.0.11 registry="git.internalnet.com:4567"
time="2024-02-08T21:29:49Z" level=debug msg="Using commit message: "
time="2024-02-08T21:29:49Z" level=info msg="Committing 2 parameter update(s) for application psa" application=psa
time="2024-02-08T21:29:49Z" level=info msg="Successfully updated the live application spec" application=psa
time="2024-02-08T21:29:49Z" level=info msg="Processing results: applications=6 images_considered=7 images_skipped=0 images_updated=2 errors=0"

This is what the manifest looks like. Even though I updated both, only one is showing up:

project: default
source:
  repoURL: '[email protected]:netops/kubernetes/k8s-apps.git'
  path: stg/psa-upon
  targetRevision: HEAD
  kustomize:
    images:
      - >-
        git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg:v1.0.12
destination:
  server: 'https://kubernetes.default.svc'
  namespace: psa
syncPolicy:
  automated:
    prune: true
    selfHeal: true
  syncOptions:
    - CreateNamespace=true

Also, here is the kustomization file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ./email-consumer-deployment.yaml
- ./sms-consumer-deployment.yaml
namespace: psa
images:
- name: git.internalnet.com:4567/software-engineers/psa/upon/email-consumer/stg
 newTag: "v1.0.1"
- name: git.internalnet.com:4567/software-engineers/psa/upon/sms-consumer/stg
 newTag: "v1.0.3"

joelgriffiths avatar Feb 08 '24 21:02 joelgriffiths

After messing with this for hours and hours, I was finally found a workaround (unless I'm wrong about why it was not working).

I believe the trick was using kustomization name to reference a tag in Deployment (not the registry) and using newName in kustomization.yaml to replace the image name in the Deployment.

kustomization.yam

kind: Kustomization
resources:
- ./email-consumer-deployment.yaml
- ./sms-consumer-deployment.yaml
namespace: psa
images:
- name: email-consumer
  newName: git.internalnet.com:4567/software-engineers/psa/email-consumer/stg
  newTag: "v1.0.8"
- name: sms-consumer
  newName: git.internalnet.com:4567/software-engineers/psa/sms-consumer/stg
  newTag: "v1.0.3"

deployment1.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sms-consumer
  namespace: psa
  labels:
    app: sms-consumer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sms-consumer
  template:
    metadata:
      labels:
        app: sms-consumer
    spec:
      containers:
      - name: sms-consumer
        image: sms-consumer
        envFrom:
        - configMapRef:
            name: sms-consumer-env-cm
        resources:
          limits:
            memory: "512Mi"
            cpu: "200m"
      imagePullSecrets:
      - name: argocd-deploy-token

deployment2.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: email-consumer
  namespace: psa
  labels:
    app: email-consumer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: email-consumer
  template:
    metadata:
      labels:
        app: email-consumer
    spec:
      containers:
      - name: email-consumer
        image: email-consumer
        envFrom:
        - configMapRef:
            name: email-consumer-env-cm
        resources:
          limits:
            memory: "512Mi"
            cpu: "200m"
      imagePullSecrets:
      - name: argocd-deploy-token

argocd-application.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: psa
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd-image-updater.argoproj.io/image-list: "sms-consumer=git.internalnet.com:4567/software-engineers/psa/sms-consumer/stg:1.0.x, email-consumer=git.internalnet.com:4567/software-engineers/psa/email-consumer/stg:1.0.x"

    # email-consumer
    argocd-image-updater.argoproj.io/email-consumer.kustomize.image-name: "email-consumer"
    argocd-image-updater.argoproj.io/email-consumer.allow-tags: "regexp:^v1\\.0\\..*$"
    argocd-image-updater.argoproj.io/email-consumer.pull-secret: pullsecret:argocd/git-software-engineers-registry-credentials
    argocd-image-updater.argoproj.io/email-consumer.update-strategy: semver

    # sms-consumer
    argocd-image-updater.argoproj.io/sms-consumer.kustomize.image-name: "sms-consumer"
    argocd-image-updater.argoproj.io/sms-consumer.allow-tags: "regexp:^v1\\.0\\..*$"
    argocd-image-updater.argoproj.io/sms-consumer.pull-secret: pullsecret:argocd/git-software-engineers-registry-credentials
    argocd-image-updater.argoproj.io/sms-consumer.update-strategy: semver

spec:
  project: default

  source:
    repoURL: [email protected]:netops/kubernetes/k8s-apps.git
    targetRevision: HEAD
    path: stg/psa
    kustomize: {}
  destination:
    server: https://kubernetes.default.svc
    namespace: psa

  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      selfHeal: true
      prune: true

joelgriffiths avatar Feb 09 '24 02:02 joelgriffiths