CLI `argocd admin cluster stats` show incorrect shard

[pjaronek]

Checklist:

• [*] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [*] I've included steps to reproduce the bug.
• [*] I've pasted the output of argocd version.

Describe the bug

I'm trying to manually assign clusters to different application-controller shards. According the logs it works as expected, each application controller does handle only its own clusters.

But argocd admin cluster stats does not show correct shard number.

To Reproduce

• setup argocd with multiple application-controller shards
• assign different clusters different shard number
• check argocd admin cluster stats output

Expected behavior

argocd-cli reports correct cluster shards/stats

Screenshots

Wrong, k3d-argocd-main set with shard: 0, k3d-argocd-test01 and k3d-argocd-test02 set with shard: 1

$ argocd admin cluster stats
SERVER                           SHARD  CONNECTION  NAMESPACES COUNT  APPS COUNT  RESOURCES COUNT
https://k3d-argocd-test01:35421  1      Successful  1                 1           13
https://k3d-argocd-test02:45027  0      Successful  1                 1           13
https://k3d-argocd-main:39443    0      Successful  1                 1           134
https://kubernetes.default.svc   0      Unknown     0                 0           0

Version

$ argocd version
argocd: v2.5.3+0c7de21
  BuildDate: 2022-11-28T17:11:59Z
  GitCommit: 0c7de210ae66bf631cc4f27ee1b5cdc0d04c1c96
  GitTreeState: clean
  GoVersion: go1.18.8
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.5.3+0c7de21
  BuildDate: 2022-11-28T16:51:33Z
  GitCommit: 0c7de210ae66bf631cc4f27ee1b5cdc0d04c1c96
  GitTreeState: clean
  GoVersion: go1.18.8
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.7 2022-08-02T16:35:54Z
  Helm Version: v3.10.1+g9f88ccb
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.18.0

Logs

Correct, k3d-argocd-main set with shard: 0, k3d-argocd-test01 and k3d-argocd-test02 set with shard: 1

$ kubectl logs argocd-application-controller-0
...
time="2022-12-02T09:57:05Z" level=info msg="Start syncing cluster" server="https://k3d-argocd-main:39443"
time="2022-12-02T09:57:05Z" level=info msg="Ignoring cluster https://k3d-argocd-test01:35421"
time="2022-12-02T09:57:05Z" level=info msg="Ignoring cluster https://k3d-argocd-test02:45027"

$ kubectl logs argocd-application-controller-1
...
time="2022-12-02T09:56:54Z" level=info msg="Ignoring cluster https://kubernetes.default.svc"
time="2022-12-02T09:56:54Z" level=info msg="Start syncing cluster" server="https://k3d-argocd-test02:45027"
time="2022-12-02T09:56:54Z" level=info msg="Start syncing cluster" server="https://k3d-argocd-test01:35421"
time="2022-12-02T09:56:54Z" level=info msg="Ignoring cluster https://k3d-argocd-main:39443"

[akram]

related to https://github.com/argoproj/argo-cd/issues/9633 .

[jon-rei]

I think this problem still exists. Even with the latest ArgoCD (v2.10.8) I can't see the correct shards assigned to my clusters:

argocd admin cluster stats
SERVER        SHARD  CONNECTION
https://xxxxx 0      Successful
https://xxxxx 0      Successful
https://xxxxx 0      Successful
https://xxxxx 0      Successful
https://xxxxx 0      Successful
https://xxxxx 0      Successful
https://xxxxx 0      Successful
https://xxxxx 0      Successful

But when I look at the application controller logs, I see the following:

level=info msg="Cluster https://xxxxx has been assigned to shard 2"
level=info msg="Cluster https://xxxxx has been assigned to shard 1"
level=info msg="Cluster https://xxxxx has been assigned to shard 0"
level=info msg="Cluster https://xxxxx has been assigned to shard 0"
level=info msg="Cluster https://xxxxx has been assigned to shard 1"
level=info msg="Cluster https://xxxxx has been assigned to shard 0"
level=info msg="Cluster https://xxxxx has been assigned to shard 1"
level=info msg="Cluster https://xxxxx has been assigned to shard 2"

[divy4]

I can confirm Jon's comment. I'm also seeing shard 0 on all clusters when using v2.10.8.

[christianh814]

@jon-rei @divy4 Are you still seeing this with 2.11+ versions?

[jon-rei]

Sorry for the late reply, with the latest version 2.12.1 I don't see the problem anymore. I think this problem is fixed.