Multiple metrics endpoints when using `expose` + multliple workers

[SebastianScherer88]

I have a fastapi app that i annotate and expose with the instrumentator, and it works as expected in that it adds a /metrics GET endpoint with all metrics. When i serve this app using more than 1 (lets say uvicorn) servers, i naturally get more than one /metrics endpoint. which of the various works i connect to when visiting {url}/metrics seems to be randomly decided by the load balancing logic of the parent worker process.

this however is an issue, as prometheus presumably would only ever connect to one randomly selected worker process' /metrics endpoint and scrape what is essentialy the nth part of the metrics it should get.

is there a way to avoid this (breaking imo) issue and consolidate the metrics resulting from exposing the app to the instrumentator into one metrics endpoint? I have seen other libraries' approach to this (e.g. kserve - they dont use this library and rely on the more low level prometheus-client library directly), but would be interested in your opinion

[tahsintahsin]

We are observing the same issue and would be interested to hear as well

[JcGKitten]

Hey, did you see this discussion. But I'm also struggling to get it to work and add more Metrics, especially ones, not instrumented by the app.

[tahsintahsin]

Hey, Thanks for pointing us to the discussion. I decided to switch to prometheus_client library and I am trying to setup for multiple processes now. It seems even there, there are limitations, but it is well explained on their side

[JcGKitten]

My current approach:

1. Set ENV variable PROMETHEUS_MULTIPROC_DIR do a directory where the metrics can be stored
2. To add custom metrics from prometheus client:
   1. create a CollectorRegistry and add it to MultiProcessCollector

       registry = CollectorRegistry()
       multiprocess.MultiProcessCollector(registry)
   
       custom_metric = Counter("test", documentation="test", registry=registry)
   

Then these metrics are exposed on the same endpoint as the ones from prometheus-fastapi-instrumentator.

Hope it helps someone :)

[trallnag]

@tahsintahsin, note that prometheus_client library is used by this library here as well. It is just a wrapper around it that provides a middleware for FastAPI and a few metrics. So the same limitations apply