Major performance issue on a very large ArgoCD instance

[Whisper40]

Checklist:

• [X] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [X] I've included steps to reproduce the bug.
• [X] I've pasted the output of argocd version.

Describe the bug

We have only one Kubernetes cluster on which ArgoCD is deployed. The ArgoCD instance has 4200 applications with 2000 repositories. At present, ArgoCD is on the verge of being usable. UI performance is catastrophic: every time we switch applications, we have to wait 2 minutes to load the "/applications" section. /repositories is also very slow (1 minute).

I'm aware that pagination of the "/applications" page is planned on the server side in a future release, but I'm worried about the real gain we'll have at all levels.

The overall health of the applications is very good.

To Reproduce

Install ArgoCD Deploy 4K+ applications and 2K+ repositories. Try to use the UI (take 2 coffee)

Expected behavior

The "idea" is to be able to use the UI without latency..

Our specific configuration

timeout.reconciliation: 360s
controller.status.processors: 100
controller.operation.processors: 50
controller.log.level: error
server.enable.gzip: true
server.log.level: error
reposerver.log.level: error

argo-cd-argocd-server-hpa : HPA (30 pods, memory limit : 12GB) argo-cd-argocd-repo-server-hpa : HPA (10 pods, memory limit : 2GB) application-controller : memory limit : 16GB Version v2.5.21+f627b62

[ericblackburn]

You are several versions behind. Before investigating, I would suggest using the latest 2.7 at the very least. They have been working on optimizing a lot and I think the UI experience was one of them.

[alexmt]

I suspect 2000 repositories might contribute to controller slowness/high memory usage. I would recommend using shared credentials instead: https://argo-cd.readthedocs.io/en/stable/user-guide/private-repositories/#credential-templates

[AnubhavSabharwa]

I am on 2.10.2 version and facing similar issue.

[Whisper40]

Actually on 2.8, and performance is really bad

[morawat]

We are observing the same behavior with 2500 applications in 2.10. is there any workaround? Is there any matrix on the CPU and memory recommendations for 3000+ apps?