APISIX EKS Prometheus Grafana

[sanhofman]

Issue description

While deploying APISIX on AWS using apisix-ingress-controller like described here: https://github.com/apache/apisix-ingress-controller/blob/master/docs/en/latest/deployments/aws.md.

We get following services up and running:

NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP                                                                  PORT(S)             AGE
apisix-admin                ClusterIP      10.100.219.157   <none>                                                                       9180/TCP            5d23h
apisix-dashboard            ClusterIP      10.100.108.89    <none>                                                                       80/TCP              5d18h
apisix-etcd                 ClusterIP      10.100.177.65    <none>                                                                       2379/TCP,2380/TCP   5d23h
apisix-etcd-headless        ClusterIP      None             <none>                                                                       2379/TCP,2380/TCP   5d23h
apisix-gateway              LoadBalancer   10.100.214.106   ...        
apisix-ingress-controller   ClusterIP      10.100.160.139   <none>                                                                       80/TCP              5d23h
prometheus-operated         ClusterIP      None             <none>                                                                       9090/TCP            5d17h

Prometheus seems up and running while accessing through port-forward (http://localhost:8090/metrics). But how do we get the URL which can be used by Grafana to read out Prometheus metrics? It seems not exposed so no external IP/URL?

Environment

• your apisix-ingress-controller version (output of apisix-ingress-controller version --long): 1.4.1
• your Kubernetes cluster version (output of kubectl version): v1.22.10

[tao12345666333]

you can refer to this one https://github.com/apache/apisix-ingress-controller/blob/master/docs/en/latest/tutorials/how-to-access-Apache-APISIX-Prometheus-Metrics-on-k8s.md

[sanhofman]

Hi @tao12345666333,

Thanks for your response.

I'm able to call the /apisix/prometheus/metrics only from within the apisix POD, but how do we make this available to Grafana dashboards?

Here we retrieve the following connection refused, both for GET and POST:

Error reading Prometheus: Post "http://localhost:9091/apisix/prometheus/metrics/api/v1/query": dial tcp 127.0.0.1:9091: connect: connection refused

[tao12345666333]

no, you should use Prometheus to fetch /apisix/prometheus/metrics

And add Prometheus as DataSource in Grafana.

[sanhofman]

Ok I'm clear with that but nevertheless I'm unable to access Prometheus from Grafana.

Available AWS services:

NAME                        TYPE           CLUSTER-IP       EXTERNAL-IP                                                                  PORT(S)             
apisix-admin                ClusterIP      10.100.219.157   <none>                                                                       9180/TCP            
apisix-dashboard            ClusterIP      10.100.108.89    <none>                                                                       80/TCP              
apisix-etcd                 ClusterIP      10.100.177.65    <none>                                                                       2379/TCP,2380/TCP   
apisix-etcd-headless        ClusterIP      None             <none>                                                                       2379/TCP,2380/TCP   
apisix-gateway              LoadBalancer   10.100.214.106   ...eu-central-1.elb.amazonaws.com   80:30114/TCP        
apisix-ingress-controller   ClusterIP      10.100.160.139   <none>                                                                       80/TCP              
grafana                     LoadBalancer   10.100.255.145   ...eu-central-1.elb.amazonaws.com   80:31666/TCP        
prometheus-operated         ClusterIP      None             <none>                                                                       9090/TCP            

While configuring Prometheus as Datasource:

[github-actions[bot]]

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions[bot]]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.