PROMETHEUS_METRICS_EXPORT_PORT_RANGE not working

[peterthomassen]

I am running a Django application with the following uwsgi configuration:

[uwsgi]
socket = [::]:3031
chdir = /usr/src/app/
wsgi-file = api/wsgi.py
processes = 64
threads = 1
uid = nobody
gid = nogroup

In settings.py, I have the following settings:

PROMETHEUS_METRICS_EXPORT_PORT_RANGE = range(8001, 8065)
PROMETHEUS_METRICS_EXPORT_ADDRESS = ''  # all addresses

According to the exports documentation in this repo, this should cause 64 ports to be opened, each exposing the metrics of one process.

When I access my views, I see that various workers handle the requests (I added debug output printing uwsgi.worker_id()). However, the only export port that gets opened is 8001, and the metrics contain only a fraction (about 20%) of the requests.

At least, the metrics counts are strictly increasing (so I suppose it's the metrics of the first process); previously (before adding the port range settings), metrics would be returned randomly from any worker.

The rest of the metrics seems to be missing, however.

Am I doing something wrong, or is this a bug?

PS: I made sure that the ports in the above range are not yet occupied.

[VinayGValsaraj]

@peterthomassen I noticed you didn't commit any changes to your prometheus.yml file regarding scraping the range of ports you provided. Prometheus must know the other ports to scrape from or else it will continue to only scrape the one port you provided.

Did you modify the config file accordingly when testing the range?

[peterthomassen]

Yes, I did modify the Prometheus config as you described, but I did not commit it as it did not work.

[asherf]

There is a special way to setup the Prometheus client library (which this project uses) to work in multi-process mode. See the docs of client docs for details I suggest using that method rather than have each process expose its own port and have Prometheus scape individual processes.

[peterthomassen]

@asherf Thanks for the suggestion, I shall give it a try. I appreciate it!

Nevertheless, this issue is about the fact that what's described here does not work as documented. Is this a bug in the software, the docs, or something else?

[jdinunzio]

Is there any news regarding this issue? I can confirm it myself.

Adding information: this happens only when gunicorn uses --preload, it doesn't happen with --reload.

[ercpe]

@peterthomassen I don't know if you found the solution to this problem, but it's important to set lazy-apps = true in the uwsgi configuration file. Otherwise, the application will be loaded into the master process before forking the child processes and the django_prometheus' app ready() function will not be called per-child but only once (and thus, only exposes one metrics endpoint).