argo-cd
argo-cd copied to clipboard
argoproj
ArgoCD metrics doesn't show in Prometheus.
Hi! I'm using the ArgoCD helm chart version: 5.17.4 and kube-prometheus-stack version: 45.27.1 and trying to expose argo metrics but even that prometheus can scrape the endpoints i can't see any metrics.
And this cause that the dashboard where query "argocd_app_info" for example, return N/A.
This extract is from server section in values.yaml
Server metrics service configuration
metrics: # -- Deploy metrics service enabled: true service: # -- Metrics service annotations annotations: {} # annotations: # prometheus.io/path: /metrics # prometheus.io/port: "8083" # prometheus.io/scrape: "true" # -- Metrics service labels labels: {} # -- Metrics service port servicePort: 8083 # -- Metrics service port name portName: http-metrics serviceMonitor: # -- Enable a prometheus ServiceMonitor enabled: true # -- Prometheus ServiceMonitor interval interval: 30s # -- Prometheus [RelabelConfigs] to apply to samples before scraping relabelings: [] # -- Prometheus [MetricRelabelConfigs] to apply to samples before ingestion metricRelabelings: [] # -- Prometheus ServiceMonitor selector # selector: {} selector: app: kube-prometheus-stack-operator release: prometheus # prometheus: kube-prometheus
# -- Prometheus ServiceMonitor scheme
scheme: ""
# -- Prometheus ServiceMonitor tlsConfig
tlsConfig: {}
# -- Prometheus ServiceMonitor namespace
namespace: "" # monitoring
# -- Prometheus ServiceMonitor labels
additionalLabels: {}
# additionalLabels:
# release: prometheus
# -- Prometheus ServiceMonitor annotations
annotations: {}
From this, prometheus can scrape ok but i don't understand why doesn't show the metrics.
Thanks an advance for any help that you can give me!!!
I had same issue when scraping from the k8s service, but if u scrape from the k8s pods it works
I'm seeing the same thing using image tag 2.9.0-f978e04c (installing using kustomize). I enable the serviceMonitors and seeing no argocd specific metrics.
I'm having the same issue with chart version 5.45.2, app version v2.8.2
I can see that the servicemonitors have deployed properly and Prometheus is hitting them
But It seems like when I manually hit the /metrics endpoint of any of the ArgoCD services, none of the Argo specific stats are returned, just what appear to be generic kube stats
$ curl http://10.99.23.87:8082/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.689e-05
go_gc_duration_seconds{quantile="0.25"} 5.481e-05
...
$ curl -s http://10.99.23.87:8082/metrics | grep argo
$
It's the same issue if I hit the service endpoint rather than the pod directly, as well
$ curl http://argo-cd-argocd-application-controller-metrics.argocd.svc.cluster.local:8082/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 3.689e-05
go_gc_duration_seconds{quantile="0.25"} 5.481e-05
...
$ curl -s http://argo-cd-argocd-application-controller-metrics.argocd.svc.cluster.local:8082/metrics | grep argo
$
It seems like the Argo stats just aren't being published at all
Interestingly, they seem to work in our production cluster. The main difference there is we are running kube 1.27 in staging and 1.23 in prod currently. We are scheduled to upgrade prod this weekend, I will follow up if that impacts this.
Hello Guys, Im facing the same issue with ArgoCD Helm Chart 5.41.2. Prometheus get all the Endpoints, but no "argocd_xxxxxxxx" metrics are available when I scrape manually. Grafana also gets nothing. Im running Kubernetes 1.26.6.
Has someone has any update on this issue? Your help is highly appreciated.
Regards,
Diego
Hi. Having the same problem on ArgoCD 2.8.4 installed from helm chart 5.46.7. K8s version: 1.28.1. No argo-specific metrics visible after manually connecting to the metrics service.
EDIT: Acctually I managed to resolve the issue myself. It turns out that ArgoCD server and app-controller starts serving metric the moment when the 1st application gets deployed. It was a brand new cluster without any workloads, so the metrics were missing because of that. I think it would be beneficial to add this information to official Docs.
Having the same issue on
- ArgoCD 2.8.4
- chart 5.46.8
- EKS 1.24.17 and 1.28.2
Directly accessing the pods metrics port returns the argocd metrics, but not via the service
EDIT: Acctually I managed to resolve the issue myself. It turns out that ArgoCD server and app-controller starts serving metric the moment when the 1st application gets deployed. It was a brand new cluster without any workloads, so the metrics were missing because of that. I think it would be beneficial to add this information to official Docs.
@Matelin That did not work for me. :(
Directly accessing the pods
metricsport returns the argocd metrics, but not via the service
In my case neither service nor endpoints had shown any argocd-specific metrics. Looks like your issue is more like @tooptoop4
I'm using helm chart version 5.50.0 and I have the same problem. I can see the ServiceMonitor is being created ...
» kubectl get servicemonitors -n argocd
NAME AGE
argo-cd-argocd-application-controller 3h31m
... however Prometheus does not have any argocd_* metric. Metrics from other projects ServiceMonitors work fine.
Here is my values.yaml:
applicationSet:
enabled: false
controller:
metrics:
enabled: true
serviceMonitor:
enabled: true
dex:
enabled: false
notifications:
enabled: false
server:
extraArgs:
- --insecure
while install argocd with helm, try add selector in values file, for the metrics section like this example(with your prometheus release):
metrics:
enabled: true
serviceMonitor:
enabled: true
interval: 30s
selector:
release: kube-prometheus-stack
I think the official docs are incorrect regarding the ServiceMonitors: I got it working; there seems to be a mismatch in configuration.
I think the official docs](https://argo-cd.readthedocs.io/en/stable/operator-manual/metrics/) should be updated. As they seem to be pointing the ServiceMonitor directly to the argocd-repo-server, rather than the argocd-repo-server-metrics, which is created when serviceMonitor is enabled within the helm-chart. The argocd-repo-server doesn't have a metrics port.
# (incorrect) from docs:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: argocd-repo-server-metrics
labels:
release: prometheus-operator
spec:
selector:
matchLabels:
app.kubernetes.io/name: argocd-repo-server
endpoints:
- port: metrics
They advice the metrics port, but it should be the http-metrics port.
Also, it's assumed that the scraping-path is /metrics.
So,
A) make sure to enable the ServiceMonitor with the helm charts,
B) make sure to set the correct path (/metrics) + port (http-metrics) on the ServiceMonitor, as the docs aren't coherent to this.
C) make sure that the matchLabels are correct: e.g. app.kubernetes.io/name: argocd-repo-server-metrics.
In sum, this should be working:
spec:
selector:
matchLabels:
app.kubernetes.io/name: argocd-repo-server-metrics
endpoints:
- port: http-metrics
path: /metrics
EDIT: Acctually I managed to resolve the issue myself. It turns out that ArgoCD server and app-controller starts serving metric the moment when the 1st application gets deployed. It was a brand new cluster without any workloads, so the metrics were missing because of that. I think it would be beneficial to add this information to official Docs.
@Matelin Unfortunately, the way you said doesn't work ㅜㅜ
The main problem for us is that the ArgoCD chart has checks on each of the ServiceMonitors like this:
{{- if and (.Capabilities.APIVersions.Has "monitoring.coreos.com/v1") .Values.controller.metrics.enabled .Values.controller.metrics.serviceMonitor.enabled }}
We would like the check:
(.Capabilities.APIVersions.Has "monitoring.coreos.com/v1")
to be removed in future releases.
We use Victoria Metrics and it doesn't have such CRDs like in Prometheus.
There is no point in this check because we can decide ourselves when to enable or disable the ServiceMonitor. Thank you.
I think the official docs are incorrect regarding the ServiceMonitors: I got it working; there seems to be a mismatch in configuration.
I think the official docs](https://argo-cd.readthedocs.io/en/stable/operator-manual/metrics/) should be updated. As they seem to be pointing the ServiceMonitor directly to the argocd-repo-server, rather than the argocd-repo-server-metrics, which is created when serviceMonitor is enabled within the helm-chart. The argocd-repo-server doesn't have a metrics port.
# (incorrect) from docs: apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: argocd-repo-server-metrics labels: release: prometheus-operator spec: selector: matchLabels: app.kubernetes.io/name: argocd-repo-server endpoints: - port: metricsThey advice the
metricsport, but it should be thehttp-metricsport. Also, it's assumed that the scraping-path is/metrics. So, A) make sure to enable the ServiceMonitor with the helm charts, B) make sure to set the correct path (/metrics) + port (http-metrics) on the ServiceMonitor, as the docs aren't coherent to this. C) make sure that the matchLabels are correct: e.g.app.kubernetes.io/name: argocd-repo-server-metrics.In sum, this should be working:
spec: selector: matchLabels: app.kubernetes.io/name: argocd-repo-server-metrics endpoints: - port: http-metrics path: /metrics
Why does your match labels has "app.kubernetes.io/name: argocd-repo-server-metrics" instead of "app.kubernetes.io/name: argocd-repo-server" ?
The reason is perhaps the service is not listening on the metrics ports.
I tried to curl the service on the port and got a connection timeout. The service monitors also fail because of the service not reachable on the port.
sh-4.2#
sh-4.2# curl http://argocd-repo-server.argocd.svc.cluster.local:8084/metrics -v
* Trying 172.20.180.85:8084...
* connect to 172.20.180.85 port 8084 failed: Connection timed out
* Failed to connect to argocd-repo-server.argocd.svc.cluster.local port 8084 after 129774 ms: Couldn't connect to server
* Closing connection
curl: (28) Failed to connect to argocd-repo-server.argocd.svc.cluster.local port 8084 after 129774 ms: Couldn't connect to server
sh-4.2#
However, the same curl to the pod IP and port works
sh-4.2# curl http://100.64.124.135:8084/metrics -v
* Trying 100.64.124.135:8084...
* Connected to 100.64.124.135 (100.64.124.135) port 8084
> GET /metrics HTTP/1.1
> Host: 100.64.124.135:8084
> User-Agent: curl/8.3.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/plain; version=0.0.4; charset=utf-8
< Date: Thu, 01 Aug 2024 20:48:21 GMT
< Transfer-Encoding: chunked
<
# HELP argocd_git_request_duration_seconds Git requests duration seconds.
# TYPE argocd_git_request_duration_seconds histogram
argocd_git_request_duration_seconds_bucket{repo="https://github.com",request_type="fetch",le="0.1"} 0
And this is perhaps because the service never exposes the metrics endpoints
sh-4.2$ kubectl get svc,ep -n argocd
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/argocd-applicationset-controller ClusterIP 172.20.196.170 <none> 7000/TCP 78d
service/argocd-dex-server ClusterIP 172.20.10.99 <none> 5556/TCP,5557/TCP 78d
service/argocd-redis ClusterIP 172.20.202.238 <none> 6379/TCP 78d
service/argocd-repo-server ClusterIP 172.20.180.85 <none> 8081/TCP 78d
service/argocd-server ClusterIP 172.20.68.159 <none> 80/TCP,443/TCP 78d
service/configmap-plugin ClusterIP 172.20.147.48 <none> 80/TCP 21d
NAME ENDPOINTS AGE
endpoints/argocd-applicationset-controller 100.64.61.103:7000 78d
endpoints/argocd-dex-server 100.64.65.38:5557,100.64.65.38:5556 78d
endpoints/argocd-redis 100.64.51.69:6379 78d
endpoints/argocd-repo-server 100.64.124.135:8081 78d
endpoints/argocd-server 100.64.91.62:8080,100.64.91.62:8080 78d
Sharing an update to anyone who might be facing similar issue, this isn't a bug. I didn't enable the metrics service by using these values in my Helm chart
applicationSet:
metrics:
enabled: true
controller:
metrics:
enabled: true
repoServer:
metrics:
enabled: true
server:
metrics:
enabled: true
Once you set these, you should see new metrics services created and your service monitors or pod monitors should work fine.
sh-4.2$ kubectl get svc,ep -n argocd
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/argocd-application-controller-metrics ClusterIP 172.20.140.86 <none> 8082/TCP 22m
service/argocd-applicationset-controller ClusterIP 172.20.67.156 <none> 7000/TCP 89d
service/argocd-applicationset-controller-metrics ClusterIP 172.20.86.240 <none> 8080/TCP 22m
service/argocd-dex-server ClusterIP 172.20.64.11 <none> 5556/TCP,5557/TCP,5558/TCP 89d
service/argocd-redis ClusterIP 172.20.226.70 <none> 6379/TCP 89d
service/argocd-redis-metrics ClusterIP None <none> 9121/TCP 22m
service/argocd-repo-server ClusterIP 172.20.145.148 <none> 8081/TCP 89d
service/argocd-repo-server-metrics ClusterIP 172.20.103.84 <none> 8084/TCP 22m
service/argocd-server ClusterIP 172.20.207.202 <none> 80/TCP,443/TCP 89d
service/argocd-server-metrics ClusterIP 172.20.21.165 <none> 8083/TCP 22m
service/configmap-plugin ClusterIP 172.20.61.220 <none> 80/TCP 25d
service/myplugin ClusterIP 172.20.169.218 <none> 80/TCP 26d
NAME ENDPOINTS AGE
endpoints/argocd-application-controller-metrics 100.64.163.91:8082 22m
endpoints/argocd-applicationset-controller 100.64.61.34:7000 89d
endpoints/argocd-applicationset-controller-metrics 100.64.61.34:8080 22m
endpoints/argocd-dex-server 100.64.25.226:5558,100.64.25.226:5557,100.64.25.226:5556 89d
endpoints/argocd-redis 100.64.86.135:6379 89d
endpoints/argocd-redis-metrics <none> 22m
endpoints/argocd-repo-server 100.64.182.129:8081 89d
endpoints/argocd-repo-server-metrics 100.64.182.129:8084 22m
endpoints/argocd-server 100.64.23.67:8080,100.64.23.67:8080 89d
endpoints/argocd-server-metrics 100.64.23.67:8083 22m
endpoints/configmap-plugin 100.64.116.102:4355,100.64.137.109:4355,100.64.31.90:4355 25d
endpoints/myplugin 100.64.80.114:4355 26d