Monstache monitoring

[hubornohub]

Hi, Currently I’m working on monitoring monstache. Although it has built-in http server but doesn’t expose metrics in Prometheus exposition format. So, I made use of promhttp go library and created custom metrics based on the endpoints like /healthz, /stats etc. But I have a bit confusion here:

1. When does monstache go down? I mean when elastic cluster or mongodb going down, it doesn’t report anything. So, based on which condition can I say it’s unhealthy and test it out to create alerts.
2. And also, unless I hit the endpoint /stats, Prometheus doesn’t fetch the metrics automatically. I want to know like am I doing something wrong here?

[erodactyl]

Did you find a solution to know when Monstache goes down? The /healthz endpoint always returns ok with status code 200. https://github.com/rwynn/monstache/blob/6f03294d2caded7d49fc05776f28c6288d2c45ff/monstache.go#L4207

[Joey0538]

@erodactyl @hubornohub We have currently forked the code and built our custom prom metrics for /healthz &/stats, & to answer your questions

1. When ES or mongo goes down, we have updated /healthz to by adding a ping operation to the ES and mongo to check if its down and send 404
2. For /stats to work, you need custom prom metrics here as well with "enable-http-server=true" & "stats=true", then only you can monitor the /stats endpoint. @rwynn hope this is the right way to go about it for monitoring