waggle-dance icon indicating copy to clipboard operation
waggle-dance copied to clipboard

Add configuration to enable exposing all monitoring data through Prometheus"

Open cs-lpc opened this issue 1 year ago • 3 comments

Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. For example - I'm always frustrated when [...]

In our scenario, we only have Prometheus-related services for viewing monitoring data. Therefore, we migrated all monitoring data from Waggledance to Prometheus. But currently, Waggledance cannot switch from Graphite to Prometheus through configuration. Therefore, I want to implement this feature.

Describe the solution you'd like A clear and concise description of what you want to happen.

Get prometheusMeterRegistry if needed in MonitoredAspect.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered. None

Additional context Add any other context or screenshots about the feature request here.

cs-lpc avatar Sep 06 '23 09:09 cs-lpc

WaggleDance exposes metrics to Prometheus too :

Link: https://github.com/ExpediaGroup/waggle-dance/blob/main/waggle-dance-core/pom.xml#L111

You will be able to find the metrics if you run curl -v localhost:18000/prometheus/actuator

Kindly let me know if you need anything else

DhrubajyotiSadhu avatar Sep 07 '23 09:09 DhrubajyotiSadhu

Thanks for you reply, I just tested it and found that after the first request, request metrics are also exposed to Prometheus. However, I noticed that these metrics do not include p75 and p90, which exist in metastore. Can I implement them?

cs-lpc avatar Sep 07 '23 09:09 cs-lpc

Would it be possible to do this on Prometheus side https://prometheus.io/docs/practices/histograms/ We'll be more flexible that way if someone want a different percentile.

patduin avatar Sep 07 '23 12:09 patduin