Store metrics if service is a scale to more 1 instances

[evgmoskalenko]

Hi,

I have a problem collecting and displaying metrics when I scale my service to 3 instances from one (for example).

The metric is stored in the service directory - "/tmp/prometheus-metrics". If there are more than one service instances, then in each service instance the metric value can be different at the time of the endpoint (/metrics) survey via load balancer:

• first instance of service (counter value = 570552): django_http_requests_total_by_view_transport_method_total{method="GET",transport="http",view="services.views.Account"} 570552
• second instance of service (counter value = 572251): django_http_requests_total_by_view_transport_method_total{method="GET",transport="http",view="services.views.Account"} 572251
• third instance of service (counter value = 560110): django_http_requests_total_by_view_transport_method_total{method="GET",transport="http",view="services.views.Account"} 560110

There is any solution to this problem? I see only one solution - service discovery and automatically updates the Prometheus job targets but it is not convenient

[asherf]

Hi @evgmoskalenko this issue is not specific to this library. I suggest looking at the Prometheus client docs or posting a question to the Prometheus Users group

Having said that, if this is not the regular gunicorn multiprocess mode but different instances (on different containers/machines) you will need configure scarping for each of those that goes directly to the container and doesn't go thru the "service" which acts as a load balancer. If by any chance you are Kubernetes, consider using the Prometheus-Operator and then configuring a PodMonitor and not a ServiceMonitor.

Hope that helps.