When I use oci private repository in helm on kustomization It has a issue.

[huntedhappy]

Checklist:

• [ ] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [ ] I've included steps to reproduce the bug.
• [ ] I've pasted the output of argocd version.

Describe the bug

When I set up kustomization.yaml in my lab It was going well and so I used argo with git but it is not work. I don't know this issue is bug or not.

I set up kustomization.yaml is like below.

It hasn't been works

vi kustomization.yaml

helmCharts:

• name: nginx repo: oci://harbor-infra.huntedhappy.kro.kr/helm version: 15.4.4 releaseName: nginx namespace: nginx valuesFile: values.yaml includeCRDs: false

To Reproduce

Expected behavior

If I use public repo on helm It works. for instanace.

vi kustomization.yaml helmCharts:

• name: nginx repo: https://charts.bitnami.com/bitnami version: 15.4.4 releaseName: nginx namespace: nginx valuesFile: values.yaml includeCRDs: false

Screenshots

Version

argocd: v2.9.3+6eba5be
  BuildDate: 2023-12-01T23:24:09Z
  GitCommit: 6eba5be864b7e031871ed7698f5233336dfe75c7
  GitTreeState: clean
  GoVersion: go1.21.4
  Compiler: gc
  Platform: linux/amd64

Logs

Paste any relevant application logs here.Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: `sh -c "kustomize build --enable-helm"` failed exit status 1: Error: Error: looks like "oci://harbor-infra.huntedhappy.kro.kr/helm" is not a valid chart repository or cannot be reached: object required : unable to run: 'helm pull --untar --untardir /tmp/_cmp_server/9269bcaa-94c3-4bd2-b64a-fb6fcc7d92d4/nginx/charts --repo oci://harbor-infra.huntedhappy.kro.kr/helm nginx --version 15.4.4' with env=[HELM_CONFIG_HOME=/tmp/kustomize-helm-3756408681/helm HELM_CACHE_HOME=/tmp/kustomize-helm-3756408681/helm/.cache HELM_DATA_HOME=/tmp/kustomize-helm-3756408681/helm/.data] (is 'helm' installed?): exit status 1

[rasheedamir]

Facing the same issue.

The public chart works perfectly fine but fails on private charts

{
    "Version": "v2.9.3+6eba5be",
    "BuildDate": "2023-12-01T23:05:50Z",
    "GitCommit": "6eba5be864b7e031871ed7698f5233336dfe75c7",
    "GitTreeState": "clean",
    "GoVersion": "go1.21.3",
    "Compiler": "gc",
    "Platform": "linux/arm64",
    "KustomizeVersion": "v5.2.1 2023-10-19T20:13:51Z",
    "HelmVersion": "v3.13.2+g2a2fb3b",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.20.0"
}

[rasheedamir]

This is nothing to do with argo-cd! It's a kustomize limitation - https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/helmcharts/#long-term-support

[fandujar]

@huntedhappy @rasheedamir I'm working on a PR to workaround it. It will add a helm registry login on the kustomize manifest inflate.

I talked to @crenshaw-dev about it.

I don't know if it will cover all cases but will certainly work for private helm repositories on Google Cloud.

[paulSambolin]

@rasheedamir kustomize does support support private OCI registries, it is new and the docs are out of date. For example, we are using ECR as OCI. Locally login aws ecr get-login-password | helm registry login --username AWS --password-stdin 111111111111.dkr.ecr.us-east-1.amazonaws.com and then pull the helm chart and render the manifests kustomize build --enable-helm. Locally it uses the credentials stored in /Users/paulsambolin/Library/Preferences/helm/registry/config.json.

@cbui got it working with argo too. First edit the argocd-repo-server deployment to set readOnlyRootFilesystem: false. Then exec to the argocd-repo-server pod and create ~/.docker/config.json. This was just a test, we will be injecting credentials using a k8s secret and volume mount. Because we use ECR, the credentials expire after 24 hours and we will be automatically refreshing the secret

~/.docker/config.json on argocd-repo-server

{
        "auths": {
                "111111111111.dkr.ecr.us-east-1.amazonaws.com": {
                        "username": "AWS",
                        "password": "...."
                }
        }
}

[marcusnh]

I'm experiencing a similar issue, but the problem seems to be with authentication even though we have created a secret to authenticate towards the ACR. See my Github issue for more information

[fandujar]

Guys, I gave up on my PR. If you want to try, the problem is here https://github.com/argoproj/argo-cd/blob/65869a3860c7555b3ea7a962db44cc9b05f7e333/reposerver/repository/repository.go#L1387-L1393

We need to check if kustomization is using helmCharts plugin, then check if the repository is OCI, and make the helm registry login using the credentials used at the argocd repository creation.

c/c @crenshaw-dev

[zimmertr]

@cbui got it working with argo too. First edit the argocd-repo-server deployment to set readOnlyRootFilesystem: false. Then exec to the argocd-repo-server pod and create ~/.docker/config.json. This was just a test, we will be injecting credentials using a k8s secret and volume mount. Because we use ECR, the credentials expire after 24 hours and we will be automatically refreshing the secret

@paulSambolin In the case of AWS, one should be able to create a new IAM role with whatever permissions are necessary to read from ECR, and then use IRSA to bind that role to the Repo Server Service Account. Which would mitigate the need to do any filesystem tricks on the pods or managing expiring credentials, no?

I'll be giving this a try tomorrow... We already use IRSA with Argo to facilitate multi-cluster management. This is a well established pattern for Argo users on AWS. I'll report back my findings after.

[cbui]

one should be able to create a new IAM role with whatever permissions are necessary to read from ECR, and then use IRSA to bind that role to the Repo Server Service Account.

I don't think this works. That's what I thought initially as well. If you try it, let me know how it goes.

[zimmertr]

@cbui, you're right. Granting the following (All list/read actions to ecr and ecr-public) to the IRSA-bound role doesn't seem to enable auth/pull. Still getting a 401. Seems like helm pull is refusing to use anything but a config.json file for auth?

        {
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchGetImage",
                "ecr:BatchGetRepositoryScanningConfiguration",
                "ecr:DescribeImageReplicationStatus",
                "ecr:DescribeImageScanFindings",
                "ecr:DescribeRegistry",
                "ecr:DescribeRepositories",
                "ecr:DescribeRepositoryCreationTemplate",
                "ecr:GetAuthorizationToken",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetLifecyclePolicy",
                "ecr:GetLifecyclePolicyPreview",
                "ecr:GetRegistryPolicy",
                "ecr:GetRegistryScanningConfiguration",
                "ecr:GetRepositoryPolicy",
                "ecr:ListTagsForResource",
                "ecr:ValidatePullThroughCacheRule",
                "ecr:ListImages",
                "ecr:DescribePullThroughCacheRules",
                "ecr:DescribeImages",
                "ecr-public:DescribeImageTags",
                "ecr-public:DescribeRegistries",
                "ecr-public:DescribeRepositories",
                "ecr-public:BatchCheckLayerAvailability",
                "ecr-public:DescribeImages",
                "ecr-public:GetAuthorizationToken",
                "ecr-public:GetRegistryCatalogData",
                "ecr-public:GetRepositoryCatalogData",
                "ecr-public:GetRepositoryPolicy",
                "ecr-public:ListTagsForResource"
            ],
            "Resource": [
                "arn:aws:ecr:$REGION:$ACCOUNT:repository/$REPOSITORY"
            ],
            "Effect": "Allow"
        }

What a shame. So the options for now then are (in order of suck):

1. Not use Kustomize to inflate Helm Charts with Argo.
2. Wait for Argo or Kustomize to provide a proper auth mechanism.
3. Set up some sort of internal proxy to host the chart with internally readable privs.
4. Hack the filesystem of the Repo Server container to include a config.json file and periodically rotate the creds.

[zimmertr]

⚠️ WARNING ⚠️ This is a roundabout solution that solves a software problem with infrastructure which is seldom a good idea. If you can afford to, either stop using Kustomize or wait until Argo/Kustomize supports an improved auth model for OCI artifacts in private registries when using Kustomize.

---

I have a working solution that is automated and doesn't involve any proxies, manual filesystem tweaks, or manual credential rotation. I won't share the exact implementation, but here is the idea:

I deploy Argo CD using the provided Helm Chart which exposes the repoServer.serviceAccount, repoServer.extraContainers, repoServer.volumes, and repoServer.volumeMounts values.

Using these values:

1. Define an extraContainer that uses an IAM role with permissions granted to it by IRSA that allows it to run aws ecr get-login-password once an hour. Massage the STDOUT from that command into a JSON blob that emulates config.json. Save this file to an emptyDir volume in the container. I use a script to do this that is volume mounted in from a ConfigMap.
2. Mount the emptyDir volume again into the Application Controller and RepoServer containers at a path where Kustomize can find it. Make it ReadOnly on this side.

The first container will create, and then update, the config.json file once an hour and then save it to a volume that is shared with Application Controller and Repo Server. Kustomize, inside the containers, will then use this config file to auth to ECR and pull the OCI artifact.

Declarative, automated, no expiration, etc.

[chancez]

I hit a bunch of issues trying to get this to work. One issue is that kustomize overrides the HELM_CONFIG_HOME, so you can't simply just helm registry login anymore. In order to make it work, I found that HELM_REGISTRY_CONFIG isn't modified by kustomize, so the solution is based on using that to configure where helm looks for registry crews.

Here's my argo-cd helm values.yaml

repoServer:
  initContainers:
    # Setup an init-container to login to private docker registries with 'helm registry login'.
    # This is needed because currently,  the upstream methods of configuring
    # authentication to private helm repositories does not work with
    # kustomize's helmCharts feature and OCI chart repositories.
    - name: registry-auth
      # copied from https://github.com/argoproj/argo-helm/blob/main/charts/argo-cd/templates/argocd-repo-server/deployment.yaml
      image: '{{ default .Values.global.image.repository .Values.repoServer.image.repository }}:{{ default (include "argo-cd.defaultTag" .) .Values.repoServer.image.tag }}'
      env:
        - name: HELM_CONFIG_HOME
          value: /helm-auth
        # Configure where helm looks for OCI registry credentials.
        - name: HELM_REGISTRY_CONFIG
          value: /helm-auth/config.json
        - name: QUAY_USERNAME
          valueFrom:
            secretKeyRef:
              name: argocd-quay-helm-credentials
              key: username
        - name: QUAY_PASSWORD
          valueFrom:
            secretKeyRef:
              name: argocd-quay-helm-credentials
              key: password
      volumeMounts:
        # mount the directory that helm will use in the main container
        - mountPath: /helm-auth
          name: registry-auth-dir
      command:
        - /bin/bash
        - -exc
        - 'helm registry login quay.io --username $QUAY_USERNAME --password $QUAY_PASSWORD'
  env:
    - name: HELM_REGISTRY_CONFIG
      value: /helm-auth/config.json
  volumes:
    - name: registry-auth-dir
      emptyDir: {}
  volumeMounts:
    # contains config.json with OCI registry credentials
    - mountPath: /helm-auth
      name: registry-auth-dir
  containerSecurityContext:
    # required for helm to untar the OCI charts
    readOnlyRootFilesystem: true

[TheMatrix97]

Hi! Any updates on this issue? Thanks!

[hawkesn]

This approach worked for me: https://github.com/argoproj/argo-cd/issues/16623#issuecomment-1877669497 Mounting the ~/.docker/config.json to the repo server.

Thank you @paulSambolin!

This is what worked for me, if you are using the ArgoCD helm chart, in your values.yaml:

...
repoServer:
  containerSecurityContext:
    readOnlyRootFilesystem: false
  volumes:
    - name: dockerconfigjson
      secret:
        secretName: <name of secret>
        items:
          - key: .dockerconfigjson
            path: config.json
  volumeMounts:
    - name: dockerconfigjson
      mountPath: /home/argocd/.docker/
      readOnly: true
...

I am also using External Secrets with the ECR generator: Which auto-refreshes the token every 12 hours without needing a script.

---
apiVersion: generators.external-secrets.io/v1alpha1
kind: ECRAuthorizationToken
metadata:
  name: ecr-gen
spec:
  region: <your region>
# This pattern above is using the host machines' node iam role to auth
# You will need to adjust this to how you're authenticating, eg. IAM, aws keys, etc.
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: <name of external secret>
spec:
  refreshInterval: "1h" # Refresh interval between ES and k8s secret, not re-authing the AWS token
  target:
    name: <name of secret>
    template:
      type: kubernetes.io/dockerconfigjson
      data:
        .dockerconfigjson: |
          {
            "auths": {
              "<AWS_ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com": {
                "username": "AWS",
                "password": "{{ .password }}",
                "auth": "{{ printf "%s:%s" "AWS" .password | b64enc }}"
              }
            }
          }
  dataFrom:
  - sourceRef:
      generatorRef:
        apiVersion: generators.external-secrets.io/v1alpha1
        kind: ECRAuthorizationToken
        name: "ecr-gen"

Hope that helps!

[zimmertr]

ECRAuthorizationToken is cool! Thanks for sharing.

[kumari-shubham]

I am also not able to deploy n8n oci chart using argocd, getting the below error

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): `kustomize build <path to cached source>/staging/n8n-cluster --enable-helm` failed exit status 1: Error: Error: could not find protocol handler for: : unable to run: 'helm pull --untar --untardir <path to cached source>/staging/n8n-cluster/charts --repo 8gears.container-registry.com/library/n8n n8n --version 0.23.1' with env=[HELM_CONFIG_HOME=/tmp/kustomize-helm-1677674532/helm HELM_CACHE_HOME=/tmp/kustomize-helm-1677674532/helm/.cache HELM_DATA_HOME=/tmp/kustomize-helm-1677674532/helm/.data] (is 'helm' installed?): exit status 1

[chancez]

@kumari-shubham you need to use oci:// in your repository URL