GKE workload identity support for gcr.io

[madhukar93]

is argocd-image-updater supposed to just work:tm: with GKE workload identity for gcr.io auth? We're seeing authentication errors -

"Failed to read tags for host 'gcr.io', repository '/v2/xxx-project/xxx-api/tags/list'

will add more context if required

We've been able to make it work with service account keys, but workload identity is a superior way to authenticate against google cloud services. If workload identity isn't a part of the core, any recommendations to go about using it?

[madhukar93]

looks like it's a feature request and not a bug, and would be addressed in PRs made after - https://github.com/argoproj-labs/argocd-image-updater/pull/307 Am I correct in that assumption?

[jannfis]

We currently only support basic auth for registries. There is some demand to also provide support for token based auth (e.g. through Authorization: bearer header or cookies), but support for vendor specific AuthZ methods is difficult.

For one, we want to stay vendor neutral, and also we don't have the required resources for implementing and testing vendor specific code.

What would it take to implement GKE workload identity?

[mikesmitty]

I'm not sure what sort of implementation would be preferable, but there's a few different options that are pretty straightforward. If you grab a token with the google oauth library like so: https://github.com/sl1pm4t/gcp-exec-creds/blob/master/main.go#L52-L66 The returned workload identity token can then be used either as a regular bearer token to GCR or it can also be used as a password for basic auth with the "oauth2accesstoken" username. The tokens do expire after an hour, but they're easy enough to refresh.

Alternatively, if you wanted to remain completely vendor-neutral and you implemented a standard docker credential helper interface, there is a standalone docker-credential-gcr tool. It can be supplied by users (or added to the image, it's only ~5MB) to provide the workload identity token in the docker credential helper format: https://github.com/GoogleCloudPlatform/docker-credential-gcr

[djfinnoy]

We've been able to make it work with service account keys,

How did you do this? I can't access our private GCR repo, I've tried docker login and passing _json_key:<service-account-json-key> as basic auth credentials.

Edit: Got it working with _json_key:<service-account-json-key>. Must have made some mistake the first time i tried.

[bkanuka]

This should be possible using the script authentication

The following script ~should work to output the Workload Identity credentials. (untested - I will update when I get it fully working)~

EDIT: It works!

So I'm not going to go into details of setting up Workload Identity - but basically you need to modify the Kubernetes Service Account running argo-image-updater (default KSA is called argocd-image-updater) and annotate it with the Google Service Account you want it to use. Then add the WorkloadIdentityUser permission to the Google Service Account.

Then to set up Argocd, create the following configmap in the argocd namespace:

apiVersion: v1
kind: ConfigMap
metadata:
  name: auth-cm
data:
  auth.sh: |
    #!/bin/sh
    ACCESS_TOKEN=$(wget --header 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token -q -O - | grep -Eo '"access_token":.*?[^\\]",' | cut -d '"' -f 4)
    echo "oauth2accesstoken:$ACCESS_TOKEN"

Edit the argocd-image-updater Deployment with the following:

      volumes:
      - configMap:
          defaultMode: 0755
          name: auth-cm
        name: auth

and

volumeMounts:
- mountPath: /auth
  name: auth

The above will mount the ConfigMap into /auth on the argcd-image-updater pod, and set the permissions to be executable. Then edit the ConfigMap argocd-image-updater-config to have the following:

data:
  log.level: debug
  registries.conf: |
    registries:
    - name: GCP Artifact Registry
      prefix: us-docker.pkg.dev
      api_url: https://us-docker.pkg.dev
      credentials: ext:/auth/auth.sh
      credsexpire: 30m

And you should be good to go!

[jfmrm]

@bkanuka You are the best, thanks a lot. Worth noting that if you want to validate that it's working as stated in the documentation: https://argocd-image-updater.readthedocs.io/en/stable/install/testing/ You should pass the flag --registries-conf-path ./app/config/registries.conf.

[bkanuka]

Glad you could get it figured out! This solution has worked for me (and others) for months now, and arguably it's not a workaround, but the "recipe" of authenticating while Argo stays vendor nuetral.

If this was helpful to someone, maybe it could be added to the docs? https://argocd-image-updater.readthedocs.io/en/stable/basics/authentication/#using-a-script-to-generate-credentials

It could be an easy PR ;)

[aro5000]

@bkanuka +1 thank you for this! - just wanted to add my experience in case anyone else runs into my same issue - if you're using a regional specific artifact registry, you need to use that region's specific prefix and api_url in the configmap. For example, if you are in us-central1, the configmap should look like this:

data:
  log.level: debug
  registries.conf: |
    registries:
    - name: GCP Artifact Registry
      prefix: us-central1-docker.pkg.dev
      api_url: https://us-central1-docker.pkg.dev
      credentials: ext:/auth/auth.sh
      credsexpire: 30m

The only other piece that gave me some issues is enabling workload identity on the image updater. For some reason I needed to add automountServiceAccountToken: true in the deployment/pod spec so the token mounted successfully at /var/run/secrets/kubernetes.io/serviceaccount/token.

Anyways, your script worked perfectly. Thanks again!

[yasharne]

Hi @bkanuka I did all the steps you mentioned, I can even login with this command:

docker login -u oauth2accesstoken -p $ACCESS_TOKEN us-east1-docker.pkg.dev

But the pod still cannot connect to the registry:

time="2023-10-16T23:03:38Z" level=debug msg="Considering this image for update" alias=myimage application=my-dev1-core image_name=my-staging/dev1/core image_tag=3f7282a08ec319950560eaf72ed1109b67813a44 registry=us-east1-docker.pkg.dev
time="2023-10-16T23:03:38Z" level=debug msg="Using no version constraint when looking for a new tag" alias=myimage application=my-dev1-core image_name=my-staging/dev1/core image_tag=3f7282a08ec319950560eaf72ed1109b67813a44 registry=us-east1-docker.pkg.dev
time="2023-10-16T23:03:38Z" level=error msg="Could not get tags from registry: Get \"https://us-east1-docker.pkg.dev/v2/my-staging/dev1/core/tags/list\": denied: Permission \"artifactregistry.repositories.downloadArtifacts\" denied on resource \"projects/my-staging/locations/us-east1/repositories/dev1\" (or it may not exist)" alias=myimage application=my-dev1-core image_name=my-staging/dev1/core image_tag=3f7282a08ec319950560eaf72ed1109b67813a44 registry=us-east1-docker.pkg.dev
time="2023-10-16T23:03:38Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=1"

[bkanuka]

@yasharne

Permission "artifactregistry.repositories.downloadArtifacts" denied

Looks like you don't have sufficient privileges on the service account.

[yasharne]

@bkanuka I changed service account permission to artifactregistry.repoAdmin role, but I still get the error, Also, I run this command on a pod with argocd-image-updater's service account and I can get the list of images with gcloud command:

gcloud artifacts docker images list us-east1-docker.pkg.dev/XXX/YYY/ZZZ
Listing items under project XXX, location us-east1, repository YYY.