django-prometheus
django-prometheus copied to clipboard
korfuri
Avoid different ports after reloading uwsgi
Hello.
In my environment I'm running Django in 2 uwsgi processes and my settings.py contains
PROMETHEUS_METRICS_EXPORT_PORT_RANGE = range(8070, 8079)
When I run it the first time, ports 8070 and 8071 serve the metrics and everything works fine.
However sometimes I need to reload/restart uwsgi. It happens sometimes that I don't get the same ports allocated. Instead, I may get 8071 and 8072. This breaks my monitoring system and I get flooded in alerts, because my setup expects two processes running and exposing metrics at ports 8070 and 8071.
I'm wondering if it makes sense to wait some little time and retry in case that a port is not yet available. I have quickly made this change note: I haven't tested it and would like to know if you find any issue with this approach or would you accept a patch to have it as a feature.
https://github.com/PabloCastellano/django-prometheus/commit/7ff2e9dffd89df67db4a287ef570e04d0ec2ee1e
@PabloCastellano thanks for your confirmation.
@korfuri and @asherf please excuse me for personally tagging you, but I needed to know if you had a possible solution to this problem. As this appears to be a very common issue, in a production environment, this scenario could easily result in pseudo failures where the uWSGI worker is exporting to a different port and the Prometheus config file is attempting to scrap metrics from a static port that is no longer being used by the respective uWSGI worker, resulting in failure scenarios.
You don’t have to use this package in this way. You can just mount the views defined in this package using your own url file. This way this package is not using ports directly and that responsibility is up to your own Django app
@asherf Thanks so much for your response; but I am unable to follow your suggested solution.
To better explain the use case, my Django app has 5 uWSGI workers, and I've used PROMETHEUS_METRICS_EXPORT_PORT_RANGE= range(8001, 8011) to export the /metrics globally for each of the 5 workers, and it works as intended.
However, in this scenario, I'm scraping metrics via a Prometheus config file that defines the ports statically, but if the ports are updated to a different port and uWSGI workers begin exporting from a different port from the available port range, the static port scraping fails with a host unreachable error, which is a pseudo failure because the workers are exporting metrics but to a different port now.
So, my question is, is there a way to learn the ports of the 5 workers from which metrics are exported, so that I can update that information to target JSON, and Prometheus can then scrape metrics from defined ports using the dynamic target JSON.
@justjais if you enable the multi-process mode, then this becomes very easy. Take a look at https://github.com/korfuri/django-prometheus/blob/master/documentation/exports.md#exporting-metrics-in-a-wsgi-application-with-multiple-processes-globally to see how to get going. Add the prometheus urls to your url conf, and when you call the API, it will automatically collect the metrics from all running processes.
