Cannot get tags from Azure Container Registry

[yavorivanov-cw]

trafficstars

Describe the bug We use AKS with "integrated" ACR, we pull images without the need of pull secrets. But argocd-image-updater cannot get image tags.

time="2025-02-05T12:50:41Z" level=error msg="Could not get tags from registry: Get \"https://myregistry.azurecr.io/v2/my-app/frontend/tags/list\": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 9ec03127-1ad8-467b-8af4-749e6cf53fa6" alias=frontend application=my-app image_name=my-app/frontend image_tag=latest registry=myregistry.azurecr.io

we get the same error for the second image backend

To Reproduce Create Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  annotations:
    argocd-image-updater.argoproj.io/image-list: backend=myregistry.azurecr.io/my-app/backend:latest,frontend=myregistry.azurecr.io/my-app/frontend:latest
    argocd-image-updater.argoproj.io/frontend.helm.image-tag: frontend.image.tag
    argocd-image-updater.argoproj.io/backend.helm.image-tag: backend.image.tag
    argocd-image-updater.argoproj.io/frontend.update-strategy: digest
    argocd-image-updater.argoproj.io/backend.update-strategy: digest
spec:
  destination:
    namespace: my-app
    server: 'https://kubernetes.default.svc'
  project: my-app
  source:
    repoURL: '[email protected]:some-repo/my-app.git'
    targetRevision: dev
    path: deploy/app
    helm:
      releaseName: my-app
      parameters:
      - name: "backend.image.tag"
        value: "latest"
      parameters:
      - name: "frontend.image.tag"
        value: "latest"
      valueFiles:
          - ../app-values/values-dev.yaml
  syncPolicy:
    automated: {}

Expected behavior I would expect that argocd-image-updater would be able to find the newest image and update the tags

Additional context The only way of working was to enable Admin user in ACR and use its credentials in secret. This is not desired solution/workaround, we need to have the admin user disabled.

Version argocd-image-updater v0.15.2+abc0072

Logs Please paste any relevant logs here

[chengfang]

Looks same issue as https://github.com/argoproj-labs/argocd-image-updater/issues/550

[yavorivanov-cw]

Looks same issue as #550

I already tried the solution with the script but still get the same error as mentioned above. Maybe I am setting wrongly the client-id in the ServiceAccount patch?! There I used the same id that I assigned the AcrPull role. But still not sure if the issue is the same as the user tried to use secret

[perlun]

For reference, we are also seeing similar issues when authenticating using a "normal" container registry token created using the Azure portal; we are seeing requested access to the resource is denied\nunauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. errors in the logs. An interesting observation is that the same credentials do work with "normal" argocd, for pulling the image using imagePullSecrets in the deployment manifest. We tried both with a token with global content/read permissions and a token with content/read and metadata/read to the specific repository; none of them worked.

The only workaround we could get working was to create an admin user in ACR. Once we did this and set it up in the Kubernetes secret, things started immediately working.

I wonder if there's a generic problem with ACR in argocd-image-updater somehow. 🤔 (Does it require some authentication mechanism not supported by it?)

[yavorivanov-cw]

For reference, we are also seeing similar issues when authenticating using a "normal" container registry token created using the Azure portal; we are seeing requested access to the resource is denied\nunauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. errors in the logs. An interesting observation is that the same credentials do work with "normal" argocd, for pulling the image using imagePullSecrets in the deployment manifest. We tried both with a token with global content/read permissions and a token with content/read and metadata/read to the specific repository; none of them worked.

The only workaround we could get working was to create an admin user in ACR. Once we did this and set it up in the Kubernetes secret, things started immediately working.

I wonder if there's a generic problem with ACR in argocd-image-updater somehow. 🤔 (Does it require some authentication mechanism not supported by it?)

We moved away from argocd-image-updater but I think the problem with the tokens is that ACR does not allow tokens to read metadata through HTTP request which are used by argocd-image-updater.

[CiucurDaniel]

@perlun solution was the only one that worked for me as well. To help some future users running in this issue I will paste an example bellow. Nonetheless, this either needs to be improved if possible or to be documented.

Image Updater with Azure Container Registry

Create secret with admin CR credentials:

kubectl create secret generic acr-admin-creds \
  --from-literal=creds="acr_admin_user:your_admin_password" \
  -n argocd-image-updater

Update the config, if using helm it would look like this:

# ArgoCD Image Updater chart values.yaml

config:
  registries: 
    - name: Your Project Azure Container Registry
      api_url: https://yourproject.azurecr.io
      prefix: yourproject.azurecr.io
      ping: yes
      credentials: secret:argocd-image-updater/acr-admin-creds#creds

Then annotation part on ApplicationCRD (an example of a helm deployment):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: bgd
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: bgd=yourproject.azurecr.io/bgd
    argocd-image-updater.argoproj.io/bgd.update-strategy: newest-build
    argocd-image-updater.argoproj.io/bgd.pull-secret: secret:argocd-image-updater/acr-admin-creds#creds
    argocd-image-updater.argoproj.io/bgd.helm.image-name: bgdDeployment.image
    argocd-image-updater.argoproj.io/bgd.helm.image-tag: bgdDeployment.tag

Future action on this issue?

I think it would be ideal if this is documented. At least to have some examples or known limitations for the major cloud providers. Maybe using the admin credentials is not allowed for some teams due to security concerns.

Another option is to improve this and find ways to use image updater without using the admin credentials of ACR if possible.

[alekc]

Some additional links which might be relevant

https://github.com/Azure/acr/issues/676 https://github.com/argoproj-labs/argocd-image-updater/issues/204 https://azure.github.io/acr/Token-BasicAuth.html#using-the-token-api