Feat/enable snapshot counts for observation

[alswl]

Problems:

When I initialize argo controller in huge cluster(more then 10000 nodes), it will takes about 1 hour to get ready.

I can only observe the progress by GetClusterInfo, but this function is blocking by a lock, so I cannot get the verbose progress.

By using this GetClusterInfoInstant, I can found how many resources is loaded by argocd controller(though this is not exactly).

Them metrics name is(with argocd metrics modified):

# HELP argocd_cluster_cache_resources_count Number of cluster cache resources
# TYPE argocd_cluster_cache_resources_count gauge
argocd_cluster_cache_resources_count{cluster="https://foo.bar:6443"} 359

# HELP argocd_cluster_cache_api_groups_count Number of cluster cache api groups
# TYPE argocd_cluster_cache_api_groups_count gauge
argocd_cluster_cache_api_groups_count{cluster="https://foo.bar:6443"} 43

PS: This is part of https://github.com/argoproj/gitops-engine/pull/350

[sonarqubecloud[bot]]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[codecov[bot]]

Codecov Report

Base: 55.38% // Head: 55.48% // Increases project coverage by +0.09% :tada:

Coverage data is based on head (ac85096) compared to base (c036d3f). Patch coverage: 100.00% of modified lines in pull request are covered.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #458      +/-   ##
==========================================
+ Coverage   55.38%   55.48%   +0.09%     
==========================================
  Files          41       41              
  Lines        4478     4506      +28     
==========================================
+ Hits         2480     2500      +20     
- Misses       1807     1813       +6     
- Partials      191      193       +2     

Impacted Files	Coverage Δ
pkg/cache/cluster.go	54.99% <100.00%> (+0.13%)	:arrow_up:
pkg/diff/diff.go	61.09% <0.00%> (+0.33%)	:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[alswl]

ping @terrytangyuan

[alswl]

@alexmt Hi, do you have time to look at this Pull Request as well?

[zachaller]

We should also add docs to https://argo-cd.readthedocs.io/en/stable/operator-manual/metrics/

[alswl]

@zachaller

We should also add docs to https://argo-cd.readthedocs.io/en/stable/operator-manual/metrics/

Yes, I created a related pull request(draft) for ArgoCD, It should be replaced by new version gitops-engine after merge.

• discuss: https://github.com/argoproj/argo-cd/issues/11157
• pr: https://github.com/argoproj/argo-cd/pull/11156

[sonarqubecloud[bot]]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
0.0% Duplication

[alswl]

Friendly ping with @zachaller and @alexmt .