Creating Application fails with 'tls: internal error'

[azaurus1]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

Most (>95%) attempts to create an application in ArgoCD fail with "remote error: tls: internal error", I've been able to atleast create an application (which doesn't sync for the same reason) by spamming the create button and one request does get through to the repo-server successfully.

I have tried all of the following:

  controller.log.level: debug
  controller.repo.server.plaintext: "true"
  reposerver.disable.tls: "true"
  reposerver.log.level: debug
  server.insecure: "true"
  server.log.level: debug
  server.repo.server.plaintext: "true"

I have also tried different Ingresses and Service Types, the error persists for all of them. As well as for attempting to create the application via CLI.

To Reproduce

1. Deploy ArgoCD to k3s cluster (even occurs in single node) via steps provided in Getting Started
2. Access the ArgoCD UI
3. Attempt to create the Guestbook application from the examples repo
4. Error should show at entering the URL in the Repository URL field

Expected behavior

Example application created successfully.

Screenshots

Version

k3s: v1.27.7+k3s1

argocd: v2.9.1+58b04e5
  BuildDate: 2023-11-14T15:40:46Z
  GitCommit: 58b04e5e11d007b0518853029ff7612c24a2eb35
  GitTreeState: clean
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/arm64
argocd-server: v2.9.1+58b04e5

Logs

repo-server:
time="2023-11-18T11:18:22Z" level=error msg="finished unary call with code Unknown" error="error testing repository connectivity: Get \"https://github.com/argoproj/argocd-example-apps.git/info/refs?service=git-upload-pack\": remote error: tls: internal error" grpc.code=Unknown grpc.method=TestRepository grpc.service=repository.RepoServerService grpc.start_time="2023-11-18T11:18:22Z" grpc.time_ms=39.377 span.kind=server system=grpc


argocd-server:
time="2023-11-18T11:18:22Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = application spec for guestbook is invalid: InvalidSpecError: repository not accessible: repositories not accessible: &Repository{Repo: \"https://github.com/argoproj/argocd-example-apps.git\", Type: \"\", Name: \"\", Project: \"\"}: repo client error while testing repository: rpc error: code = Unknown desc = error testing repository connectivity: Get \"https://github.com/argoproj/argocd-example-apps.git/info/refs?service=git-upload-pack\": remote error: tls: internal error" grpc.code=InvalidArgument grpc.method=Create grpc.service=application.ApplicationService grpc.start_time="2023-11-18T11:18:22Z" grpc.time_ms=113.037 span.kind=server system=grpc

[haidangbk]

Have you found the solution yet?

[azaurus1]

@haidangbk sorry for late response, no I haven't found a solution for this yet unfortunately

[jgwest]

Based on the error messages, and the lack of similar reports, this appears to be an issue with the specific cluster it is installed on, rather than an issue with Argo CD.

Since Argo CD CI build itself uses k3s to run E2E tests, I wouldn't expect it to be a k3s issue, for example.

[azaurus1]

Thanks @jgwest

I have resolved this by:

1. Uninstalling tailscale from all the cluster nodes.
2. Re-installing k3s
3. Re-installing ArgoCD

Notes: The cluster nodes were connected to a Headscale server, I noticed that the IP address in the error that occured while the server was running /refs while fetching application repos, was the server address of my Headscale server.

CoreDNS also had the following logs: [ERROR] plugin/errors: 2 argocd-repo-server.default.svc.cluster.local.[HEADSCALE DOMAIN] A: read udp 10.42.0.5:47256->[local gateway ip]:53: i/o timeout, which would have come from the search entry in DNS.

[half2me]

@azaurus1 I'm having the same issue trying to install tailscale on TrueNAS Scale which uses k3s. Unfortunately reinstalling k3s is not an option for me. I've tried reinstalling tailscale but the issue is the same. I'm fairly certain there is something wrong with my cluster, but all my other apps work just fine. I wonder if there is some way I can get this to work without reseting the entire k3s cluster.

[azaurus1]

@half2me are you also using headscale for the control server for the tailnet or are you using tailscales control server?

For k3s, there is the option for intercluster communications over tailscale, if that is what you are trying out: https://docs.k3s.io/installation/network-options#integration-with-the-tailscale-vpn-provider-experimental

or, if its for exposing the applications, you could also try a tailscale sidecar container: https://tailscale.com/kb/1185/kubernetes#sample-sidecar