Unable to connect to repository: rpc error

[chandanpjain]

Describe the bug Connection to helm OCI repo in AWS ECR fails to connect frequently.

To Reproduce

1. We are connecting to a helm repo on ECR and we are configuring the repo and credentials for it via helm chart deployment of argocd.
2. Here is our configuration in argocd helm values:

   repositories:
      foo-ecr-helm:
        name: foo-ecr-helm
        type: helm OCI
        url: <AWS_ACCOUNT>.dkr.ecr.<REGION>.amazonaws.com

    credentialTemplates:
      foo-ecr-helm:
        url: <AWS_ACCOUNT>.dkr.ecr.<REGION>.amazonaws.com
        username: AWS
        password: password
        enableOCI: true
        type: helm
        name: foo-ecr-helm
        plaintext: true

3. We can see helm deployment is creating the repository and its corresponding kubernetes secret for this repository.

Expected behavior

• helm repository is connected and stays in connected state.

Actual behavior

• helm repository connects sometimes
• Most of the times repo remains in disconnected (failed) state
• No clear logs to indicate the issues

Screenshots

Version

argocd: v2.10.6+d504d2b.dirty
  BuildDate: 2024-04-05T03:31:58Z
  GitCommit: d504d2b1d92f0cf831a124a5fd1a96ee29fa7679
  GitTreeState: dirty
  GoVersion: go1.22.2
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.10.7+b060053
  BuildDate: 2024-04-15T08:45:08Z
  GitCommit: b060053b099b4c81c1e635839a309c9c8c1863e9
  GitTreeState: clean
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.14.3+gf03cc04
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0

Logs We are not able to locate any logs in argocd-repo-server other than the below

repo-server time="2024-04-30T01:29:35Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:<AWS_ACCOUNT>.dkr.ecr.us-west-2.amazonaws.com,Path:,TargetRevision:7.3.0,Helm:&ApplicationSourceHelm{ValueFiles:[$values/argo-cd-applications/cust-abc/values_dev.yaml],Parameters:[]HelmParameter{},ReleaseName:,Values:,FileParameters:[]HelmFileParameter{},Version:,PassCredentials:true,IgnoreMissingValueFiles:false,SkipCrds:false,ValuesObject:nil,},Kustomize:nil,Directory:nil,Plugin:nil,Chart:foo-chart,Ref:,}/7.3.0"                                                                                                                                                                                                          
repo-server time="2024-04-30T01:29:35Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:<AWS_ACCOUNT>.dkr.ecr.us-west-2.amazonaws.com,Path:,TargetRevision:7.3.0,Helm:&ApplicationSourceHelm{ValueFiles:[$values/argo-cd-applications/customer-xyz/values_dev.yaml],Parameters:[]HelmParameter{},ReleaseName:,Values:,FileParameters:[]HelmFileParameter{},Version:,PassCredentials:true,IgnoreMissingValueFiles:false,SkipCrds:false,ValuesObject:nil,},Kustomize:nil,Directory:nil,Plugin:nil,Chart:foo-chart,Ref:,}/7.3.0"

[avo-sepp]

~We started having this problem recently. The OCI registry connection has been in-place for a long while now, but started kicking up this error today. We updated to 2.11.0 yesterday but it is not confirmed if that is related.~ Our error ended up being unrelated. Carry on.

[chandanpjain]

Suggestions on https://github.com/argoproj/argo-helm/issues/2601 helped solve this. Closing this ticket.