fusion-cloud-native icon indicating copy to clipboard operation
fusion-cloud-native copied to clipboard

Published 20 hours ago • verified publisher icon lucidworks

pulsar-bookkeeper is not exporting metrics

Open leandro-ciandt opened this issue 4 years ago • 4 comments

What is the issue?

  • the 'customize_fusion_values.yaml' example is showing that for 'pulsar-bookkeeper' we have the default endpoint '/metrics' on port '8000' for monitoring, but the 'pulsar-bookkeeper' service is not exposing that port, it's only exposing 'client:3181' port.

What is the problem?

  • on Prometheus targets i'm not seeing the '/metrics' endpoint 'UP' and the metrics are not being got by the scraper.

What is expected?

  • have the port '8000' exposed on 'pulsar-bookkeeper' service for '/metrics' endpoint and Prometheus targets reaching the metrics responses and labels.

--

p.s.: for all other five endpoints (api-gateway, indexing, pulsar-broker, query-pipeline and solr), the metrics are being scrapped fine.

leandro-ciandt avatar Apr 09 '20 15:04 leandro-ciandt

Hi @leandro-ciandt, I can't seem to reproduce this. What cloud provider are you using to deploy fusion into?

I deployed into both EKS and GKE and have bookkeeper metrics appear correctly in prometheus in both environments.

ian-thebridge-lucidworks avatar Apr 14 '20 15:04 ian-thebridge-lucidworks

hi @ian-thebridge-lucidworks, i'm using my own kubernetes cluster. i'm not using EKS, GKE or AKS.

i noticed that for bookkeeper, the created service is not exposing port 8000 (which should be used by exporter), it's exposing just the client port (3181).

leandro-ciandt avatar Apr 14 '20 16:04 leandro-ciandt

@leandro-ciandt , okay thank you.

I see that the service only exposes 3181, but that shouldn't be a problem that stops the metrics from being scraped. Prometheus is targeting the bookkeeper pods directly and not going through the kubernetes service mechanism, so it shouldn't matter that we don't expose port 8000 in the service.

is the bookkeeper instance that you are seeing as down in the prometheus UI running and in a Ready state?

ian-thebridge-lucidworks avatar Apr 14 '20 17:04 ian-thebridge-lucidworks

@ian-thebridge-lucidworks, thanks for replying.

also i'm using my own Prometheus and i'm not deploying it with the Fusion cluster. so, i'm not using the pod's annotations to configure the targets.

i'm creating the ServiceMonitors for that and it's expected that the exporters in the pods being exposed the same way as you guys are doing for other services. otherwise, i'll need to create a specific service targeting the bookkeeper exporter for that and i don't believe that's the way.

p.s.: all the services and pods are up and running. also, if a do a port-forward to the pods i can see the metrics being generated.

leandro-ciandt avatar Apr 14 '20 17:04 leandro-ciandt