argocd-image-updater
argocd-image-updater copied to clipboard
Image not being updated
The Updater sees that there is a newer tag and seems to understand it needs to do a thing, says its doing a thing, but nothing changes. I don't see an error, but I don't know your logs well enough to know what its doing. I have a very basic install, all defaults plus config to talk to private registry which is working fine. Version: argocd-image-updater v99.9.9+f1cc8e4 Here are the logs, let me know if you need more specific info.
time="2021-04-08T19:29:45Z" level=info msg="Setting new image to tjsacs.azurecr.io/itac-web:v1.1.14" alias= application=itac-web-tst-az-prd-c01 image_name=itac-web image_tag=v1.1.13 registry=tjsacs.azurecr.io
time="2021-04-08T19:29:45Z" level=info msg="Successfully updated image 'tjsacs.azurecr.io/itac-web:v1.1.13' to 'tjsacs.azurecr.io/itac-web:v1.1.14', but pending spec update (dry run=false)" alias= application=itac-web-tst-az-prd-c01 image_name=itac-web image_tag=v1.1.13 registry=tjsacs.azurecr.io
time="2021-04-08T19:29:45Z" level=info msg="Committing 0 parameter update(s) for application itac-web-tst-az-prd-c01" application=itac-web-tst-az-prd-c01
time="2021-04-08T19:29:45Z" level=debug msg="found 7 from 7 tags eligible for consideration" image="tjsacs.azurecr.io/itac-web:v1.1.13"
time="2021-04-08T19:29:45Z" level=info msg="Setting new image to tjsacs.azurecr.io/itac-web:v1.1.14" alias= application=itac-web-tst-op-prd-c01 image_name=itac-web image_tag=v1.1.13 registry=tjsacs.azurecr.io
time="2021-04-08T19:29:45Z" level=info msg="Successfully updated image 'tjsacs.azurecr.io/itac-web:v1.1.13' to 'tjsacs.azurecr.io/itac-web:v1.1.14', but pending spec update (dry run=false)" alias= application=itac-web-tst-op-prd-c01 image_name=itac-web image_tag=v1.1.13 registry=tjsacs.azurecr.io
time="2021-04-08T19:29:45Z" level=info msg="Committing 0 parameter update(s) for application itac-web-tst-op-prd-c01" application=itac-web-tst-op-prd-c01
time="2021-04-08T19:29:45Z" level=info msg="Successfully updated the live application spec" application=itac-web-tst-test2-op
time="2021-04-08T19:29:45Z" level=info msg="Successfully updated the live application spec" application=itac-web-tst-az-prd-c01
time="2021-04-08T19:29:45Z" level=info msg="Successfully updated the live application spec" application=itac-web-tst-op-prd-c01
time="2021-04-08T19:29:45Z" level=info msg="Processing results: applications=3 images_considered=3 images_skipped=0 images_updated=3 errors=0"```
I did some additional poking, the 3 applications were deployed using an appset specifically the list generator with 3 clusters. I pulled one out of the appset and deployed it like a regular app and that worked. So, what do I need to do to make this work with apps deployed with as applicationsets?
Hey @travis-sobeck, the Image Updater - in its default configuration - will change application parameters on the Application
CRs it updates, effectively changing the Application
's spec. I can just guess here (because I haven't used AppSet yet), but I'd believe that ApplicationSet will reconcile the Application
spec to the version it considers to be the correct one from the templates used. This is a somewhat similar behavior to when you'd use an app-of-apps pattern, and the Application
resource would be reconciled back from Git by Argo CD.
I think in this case, your best choice would be to use the Git write-back method to persist the changes from Image Updater in Git.
@jannfis I can test that, when I looked over the write-back options the 'argocd' option seemed a lot simpler. I can test out the 'git' method. That being said it would be nice if the these two projects were more compatible .... (unless I'm missing something)
That being said it would be nice if the these two projects were more compatible .... (unless I'm missing something)
I do fully agree with that! We'll see how we can come up with a solution that makes interoperability between the projects work better.
@jannfis maybe an addition to the annotation saying ... hey this is an AppSet, named <appSet> .. so spec looks like blah blah ..
@jannfis so it created the 2 .argocd-source
Contents of both files:
images:
- tjsacs.azurecr.io/itac-web:v1.1.14````
Hey @travis-sobeck, I don't know whether it got lost in formatting, but the contents really should be:
kustomize:
images:
- image1
- ...
Also, are you using Argo CD 2.0? Because the Git write-back has a hard-dependency to a new feature that was released with Argo CD 2.0
Oh shoot, I'm on v1.8.7, I'll upgrade monday. I just installed the latest a few days ago and though "oh sure i'm on the latest .. " haha
Hello,
I'll barge in on this ticket because I have exactly the same thing, with the exception that in my case it's a single basic Application (Helm application).
time="2021-04-12T07:19:38Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2021-04-12T07:19:38Z" level=info msg="Setting new image to gcr.io/<myregistry>/testapi:1.0.1" alias= application=mytestapi image_name=<myregistry>/testapi image_tag=1.0.0 registry=gcr.io
time="2021-04-12T07:19:38Z" level=info msg="Successfully updated image 'gcr.io/<myregistry>/testapi:1.0.0' to 'gcr.io/<myregistry>/testapi:1.0.1', but pending spec update (dry run=false)" alias= application=mytestapi image_name=<myregistry>/testapi image_tag=1.0.0 registry=gcr.io
time="2021-04-12T07:19:38Z" level=info msg="Committing 0 parameter update(s) for application mytestapi" application=mytestapi
time="2021-04-12T07:19:39Z" level=info msg="Successfully updated the live application spec" application=mytestapi
time="2021-04-12T07:19:39Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=1 errors=0"
That's about all that happens normally when the update method is the default. The application isn't updated. If I change it to git, I get the following extra events:
time="2021-04-12T07:23:39Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2021-04-12T07:23:39Z" level=info msg="Setting new image to gcr.io/<myregistry>/testapi:1.0.1" alias= application=mytestapi image_name=<myregistry>/testapi image_tag=1.0.0 registry=gcr.io
time="2021-04-12T07:23:39Z" level=info msg="Starting configmap/secret informers"
time="2021-04-12T07:23:39Z" level=info msg="Successfully updated image 'gcr.io/<myregistry>/testapi:1.0.0' to 'gcr.io/<myregistry>/testapi:1.0.1', but pending spec update (dry run=false)" alias= application=mytestapi image_name=<myregistry>/testapi image_tag=1.0.0 registry=gcr.io
time="2021-04-12T07:23:39Z" level=info msg="Committing 0 parameter update(s) for application mytestapi" application=mytestapi
time="2021-04-12T07:23:40Z" level=info msg="Configmap/secret informer synced"
time="2021-04-12T07:23:40Z" level=info msg="configmap informer cancelled"
time="2021-04-12T07:23:40Z" level=info msg="Initializing [email protected]:<mygitrepo>.git to /tmp/git-mytestapi785214539"
time="2021-04-12T07:23:40Z" level=info msg="rm -rf /tmp/git-mytestapi785214539" dir= execID=GDZEr
time="2021-04-12T07:23:40Z" level=info msg="secrets informer cancelled"
time="2021-04-12T07:23:40Z" level=info msg="git fetch origin --tags --force" dir=/tmp/git-mytestapi785214539 execID=3JDFG
time="2021-04-12T07:23:40Z" level=error msg="`git fetch origin --tags --force` failed exit status 128: No ECDSA host key is known for gitlab.com and you have requested strict checking.\r\nHost key verification failed.\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." execID=3JDFG
time="2021-04-12T07:23:40Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=1"
time="2021-04-12T07:23:40Z" level=error msg="Could not update application spec: `git fetch origin --tags --force` failed exit status 128: No ECDSA host key is known for gitlab.com and you have requested strict checking.\r\nHost key verification failed.\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists." application=mytestapi
I'm running ArgoCD v2.0.
While these errors seem pretty information, I'm wondering a few things:
- does image-updater need different known_hosts to access Git repo via SSH? It seems to try to use the same credentials used to setup the application and the application itself has no issues when it comes to sync-ing. Is there a way for me to manually provide this? I know there's a configmap for ArgoCD but it has correct key verification information for Gitlab.
- do I need to create the .argocd ? The docs seem to suggest the image updater will do that.
Thanks
@andrei-dascalu I think it's not yet documented but if you want to use SSH git repositories you need to provide .ssh/known_hosts file. For example if you installed Image Updater in ArgoCD's namespace:
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-image-updater
spec:
template:
spec:
volumes:
- name: ssh-known-hosts
configMap:
name: argocd-ssh-known-hosts-cm
containers:
- name: argocd-image-updater
volumeMounts:
- mountPath: /app/config/ssh
name: ssh-known-hosts
Upgrading to Argo 2.0 has resolved the issue, at least as far as the 'git" write back which in retrospect i do like having the permanent record in the repo of the last tag and an easy audit trail .. I have not tested with the 'argo' write-back.
Also for me, the SSH known files addition has worked! Thanks!
Upgrading to Argo 2.0 has resolved the issue, at least as far as the 'git" write back which in retrospect i do like having the permanent record in the repo of the last tag and an easy audit trail .. I have not tested with the 'argo' write-back.
Thanks for confirmation, @travis-sobeck. I think you might have no luck just yet using the argocd
write-back method in combination with ApplicationSet
, because it will probably require also a change on the AppSet side. There are already some issues in the AppSet, e.g. https://github.com/argoproj-labs/applicationset/issues/186
@jannfis
Hi. My application is not updated although the logs is showing Successfully updated the live application spec
.
I am trying to change the image every time the latest has built but I don't know what I have missed.
P.S. I am using ECR and I have written a script to get the credentials.
Application.yaml
argocd-image-updater.argoproj.io/image-list: my-app=xxx.dkr.ecr.ap-southeast-1.amazonaws.com/my-app:latest
argocd-image-updater.argoproj.io/my-app.update-strategy: digest
argocd-image-updater.argoproj.io/my-app.pull-secret: ext:/tmp/script.sh
registries.conf:
registries.conf: |
registries:
- name: ECR
api_url: https://xxx.dkr.ecr.ap-southeast-1.amazonaws.com
prefix: xxx.dkr.ecr.ap-southeast-1.amazonaws.com
ping: no
tagsortmode: none
credentials: ext:/tmp/script.sh
Logs in image-updater:ccccc
time="2021-07-06T04:26:18Z" level=info msg=/tmp/script.sh dir= execID=Cgvft
time="2021-07-06T04:26:20Z" level=info msg="Setting new image to xxx.dkr.ecr.ap-southeast-1.amazonaws.com/my-app@sha256:58a17de6205060528f56fb735c81619bec5744bd074959ca0770ddf6ad5f3365" alias=my-app application=my-app image_name=my-app image_tag=dummy registry=xxx.dkr.ecr.ap-southeast-1.amazonaws.com
time="2021-07-06T04:26:20Z" level=info msg="Successfully updated image 'xxx.dkr.ecr.ap-southeast-1.amazonaws.com/my-app@dummy' to 'xxx.dkr.ecr.ap-southeast-1.amazonaws.com/my-app@sha256:58a17de6205060528f56fb735c81619bec5744bd074959ca0770ddf6ad5f3365', but pending spec update (dry run=false)" alias=my-app application=my-app image_name=my-app image_tag=dummy registry=xxx.dkr.ecr.ap-southeast-1.amazonaws.com
time="2021-07-06T04:26:20Z" level=info msg="Committing 0 parameter update(s) for application my-app" application=my-app
time="2021-07-06T04:26:20Z" level=info msg="Successfully updated the live application spec" application=my-app
time="2021-07-06T04:26:20Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=1 errors=0"
I have same issue with you @ivanpang-crypto. Did you resolve it?
there is any way that makes this works with applicaitonset? I'm not sure why components from the same project fail to work together, argocd its cool but applicationset, notifications, image updater, rollouts all kind of fail with applicationset wich make is unusable, its frustrating
I've been using the image updater with app sets for quite some time now. I just had to make sure I was running the latest versions of everything and use the get right back method and everything seems to work.
On Thu, Sep 23, 2021, 4:42 PM Yasmany Cubela Medina < @.***> wrote:
there is any way that makes this works with applicaitonset? I'm not sure why components from the same project fail to work together, argocd its cool but applicationset, notifications, image updater, rollouts all kind of fail with applicationset wich make is unusable, its frustrating
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/argoproj-labs/argocd-image-updater/issues/186#issuecomment-926184808, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEZKYLNW2EDFF5FRTCI6YVLUDONODANCNFSM42TQ5YGA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
it seems that only works with write back but I want to use imperative vs declarative
That was my initial approach as well, but after switching to the git write-back method I found I liked having a permanent record in my git repo of exactly what changed, when, and what the current version of an image is. It also makes it easier for anyone else to see without needing access to argo. I figure the git repo should be the source of truth anyways.
On Thu, Sep 23, 2021 at 8:42 PM Yasmany Cubela Medina < @.***> wrote:
it seems that only works with write back but I want to use imperative vs declarative
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/argoproj-labs/argocd-image-updater/issues/186#issuecomment-926281436, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEZKYLJXRTV5U5N6OCP2Q3TUDPJRJANCNFSM42TQ5YGA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
i configure the write-back method to git but now it keeps recreating my pods, example of argocd logs
time="2021-09-28T13:11:47Z" level=info msg="Initialized new operation: {&SyncOperation{Revision:d33baeaa80095e8d859b4d298f2677985134151a,Prune:true,DryRun:false,SyncStrategy:nil,Resources:[]SyncOperationResource{SyncOperationResource{Group:apps,Kind:Deployment,Name:user-register-test-listener,Namespace:,},},Source:nil,Manifests:[],SyncOptions:[CreateNamespace=true PrunePropagationPolicy=foreground],} { true} [] {5 nil}}" application=user-cmd-unstable-register-user-test
time="2021-09-28T13:11:47Z" level=info msg="Ignore status for CustomResourceDefinitions"
time="2021-09-28T13:11:47Z" level=info msg="Ignore '/spec/preserveUnknownFields' for CustomResourceDefinitions"
time="2021-09-28T13:11:47Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: default)" application=user-cmd-unstable-register-user-test
time="2021-09-28T13:11:47Z" level=info msg="getRepoObjs stats" application=eck build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=345 unmarshal_ms=345 version_ms=0
time="2021-09-28T13:11:48Z" level=info msg="getRepoObjs stats" application=user-cmd-unstable-register-user-live build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=4071 unmarshal_ms=4071 version_ms=0
time="2021-09-28T13:11:48Z" level=info msg="getRepoObjs stats" application=user-cmd-unstable-register-user-test build_options_ms=0 helm_ms=0 plugins_ms=0 repo_ms=0 time_ms=4081 unmarshal_ms=4081 version_ms=0
time="2021-09-28T13:11:48Z" level=error msg="Failed to initiate auto-sync to d33baeaa80095e8d859b4d298f2677985134151a: rpc error: code = FailedPrecondition desc = another operation is already in progress" application=user-cmd-unstable-register-user-test
time="2021-09-28T13:11:48Z" level=error msg="Failed to initiate auto-sync to d33baeaa80095e8d859b4d298f2677985134151a: rpc error: code = FailedPrecondition desc = another operation is already in progress" application=user-cmd-unstable-register-user-live
time="2021-09-28T13:11:48Z" level=info msg="Update successful" application=user-cmd-unstable-register-user-live
time="2021-09-28T13:11:48Z" level=info msg="Reconciliation completed" application=user-cmd-unstable-register-user-live dedup_ms=0 dest-name= dest-namespace=default dest-server="https://kubernetes.default.svc" diff_ms=25 fields.level=824814551928 git_ms=4071 health_ms=4 live_ms=1 settings_ms=0 sync_ms=0 time_ms=4309
it seems that argocd its fighting the update. any ideas?
PD: I'm using argocd, application set and image update just to be clear
@bitgandtter , It's exactly as on our side. ImageUpdater is applying the helm change and them argo is setting it back to default when using name write back method
I'm currently facing the same issue. I have Application, and my setup is similar to what is described by @ivanpang-crypto above. I'm using argocd
as the write-back method (the default).
I see there are events in the Application resource that indicates the image updater has "successfully" updated the application:
Events:
....
Normal ImagesUpdated 115s ArgocdImageUpdater Successfully updated application 'rover-platform'
And the event details suggest additional annotations updated for the images, like so:
"argocd-image-updater.image-16/full-image-name": "xxx.dkr.ecr.us-west-2.amazonaws.com/rover-rest-service",
"argocd-image-updater.image-16/image-name": "rover-rest-service",
"argocd-image-updater.image-16/new-tag": "sha256:eaba46c42ea029b2b291aefe9f8beb9e1d4b293de7928298044069ba952bb247",
"argocd-image-updater.image-16/old-tag": "dummy",
I don't see these annotations added to the Application itself btw.
However, the image updater keep re-updating the image in each update cycle, instead of "skipping" the image since its already in-sync.
Is there anything missing here? Can anyone suggest how to resolve this issue?
I have issue with you @Dwisf . Did you resolve the problem?
@tamanh123456 :
I have issue with you @Dwisf . Did you resolve the problem?
Not yet. Still waiting for response. But I have created another ticket with more details, here: https://github.com/argoproj-labs/argocd-image-updater/issues/431
Hey @travis-sobeck, the Image Updater - in its default configuration - will change application parameters on the
Application
CRs it updates, effectively changing theApplication
's spec. I can just guess here (because I haven't used AppSet yet), but I'd believe that ApplicationSet will reconcile theApplication
spec to the version it considers to be the correct one from the templates used. This is a somewhat similar behavior to when you'd use an app-of-apps pattern, and theApplication
resource would be reconciled back from Git by Argo CD.I think in this case, your best choice would be to use the Git write-back method to persist the changes from Image Updater in Git.
I have the exact same problem as @travis-sobeck. AppSet creating Application from value file. But the container is not updated.
I have also used a git write-back to the repository and can see that proper sha256 is in the .argocd yaml file in the repository.
But it still does not update the POD spec and the container is based on the old image. :-(
Anyone actually know if the combination of AppSet and Image Updater can work together? If so, are there certain tweaks/workarounds that need to be done?
So I finally got this to work. I will leave this here for others who may run into the same issue. Note that the helm-spec did not work for me. But the following did.
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
namespace: argocd
name: my-apps
spec:
generators:
- git:
repoURL: ssh://[email protected]:7999/services/my-service.git
revision: HEAD
files:
- path: k8s/my-service/configs/dev/*-values.yaml
template:
metadata:
annotations:
argocd-image-updater.argoproj.io/image-list: '{{nameOverride}}={{image.repository}}:latest'
argocd-image-updater.argoproj.io/write-back-method: git:repocreds
argocd-image-updater.argoproj.io/git-branch: HEAD
argocd-image-updater.argoproj.io/{{nameOverride}}.update-strategy: digest
argocd-image-updater.argoproj.io/{{nameOverride}}.force-update: 'true'
argocd-image-updater.argoproj.io/{{nameOverride}}.helm.image-name: '{{nameOverride}}={{image.repository}}:latest'
argocd-image-updater.argoproj.io/{{nameOverride}}.helm.image.tag: '{{image.tag}}'
name: '{{nameOverride}}-dev'
spec:
project: argocd-availability-dev
source:
repoURL: ssh://[email protected]:7999/services/my-service.git
targetRevision: HEAD
path: 'k8s/availability-service'
helm:
releaseName: '{{nameOverride}}'
valueFiles:
- './configs/dev/{{nameOverride}}-values.yaml'
destination:
server: https://kubernetes.default.svc
namespace: '{{namespace}}'
syncPolicy:
automated:
prune: true
selfHeal: false
allowEmpty: false
syncOptions:
- CreateNameSpace=true
@TheDukeDK, your configuration only seems to work if targetRevision
= argocd-image-updater.argoproj.io/git-branch
. When git-branch
differs (unfortunately this is needed in my case because targetRevision is a protected branch), ApplicationSet seems to just reconcile reverting IU change to image. I could be misinterpreting the behavior I'm seeing but this seems to be the case.
Good Morning guys,
couple days ago I've experienced the same issue as mentioned in this forum. I had checked logs in argocd image updater pod, and I found out some messages that argocd has successfully update image in related app. however it turned out was wrong.
and here the things I've done to make it right :
- Delete all kubernetes resources owned by argocd namespace
- Delete namespace of argocd
- Reproduce argocd by creating it using this command :
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
(Note : please make sure you have created the namespace of argocd earlier) - Since the default argocd service is "ClusterIP", I decided to expose them using ingress. You can create your own ingress for your argocd accordingly.
- Edit configmap of "argocd-cm". add these following lines :
data:
resource.customizations: |
networking.k8s.io/Ingress:
health.lua: |
hs = {}
hs.status = "Healthy"
return hs
- Restart argocd deployment :
kubectl rollout restart deploy -n argocd argocd-server
- Now login to your registry account through CLI (in my case, I am using docker hub registry) :
docker login
- Create secret to store registry credential :
kubectl -n argocd create secret generic <secret_name> --from-file=.dockerconfigjson=/root/.docker/config.json --type=kubernetes.io/dockerconfigjson
- Modify configmap of "
argocd-image-updater-config
". add these following lines :
data:
log.level: debug
registries.conf: |
registries:
- name: Docker Hub
prefix: docker.io
api_url: https://index.docker.io/
ping: yes
## this is detail of secret we've created before ##
## format = pullsecret:<namespace>/<secret_name> ##
credentials: pullsecret:argocd/dockerhub-johndoe
- I am not pretty sure whether it's mandatory to do. But I suggest you to do it. kill or purge pod of argocd-image-updater everytime you make changes in its configmap.
- Access your argocd web UI using its default password and create your own app that has been created in GitHub/ GitLab earlier. (Do not enable the auto sync feature)
- Modify your app through CLI. Add these lines :
annotations:
argocd-image-updater.argoproj.io/git-branch: main
argocd-image-updater.argoproj.io/image-list: alias=docker.io/<dockerhub-account-name>/<repository-name>
argocd-image-updater.argoproj.io/alias.force-update: "true"
argocd-image-updater.argoproj.io/alias.update-strategy: latest
argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/<Git-access-token-name>
- Enable auto sync of argocd App through argocd web UI
- Wait for several minutes to take effect on it.
These are couple things I've done with my argocd within my kubernetes cluster. And I prove it on my own it works. Please do not hesitate if you guys have other questions or suggestions.
Hopefully this brief workaround could help you who experienced the same issue.
Had to add helm parameters to get it to work, see my comment here: https://github.com/argoproj-labs/argocd-image-updater/issues/431#issuecomment-1471871799
If it's related to Image updater + ApplicationSet, then there is a work arround by changing ApplicationSet controller policy. See my comment here https://github.com/argoproj/applicationset/issues/547#issuecomment-1674693159