Lack of PID uniqueness in container environments makes prometheus_multiproc difficult to use

[mkingsbury]

In environments where each container is running in its own PID namespace, multiple instances of the same container will quickly result in (silently) conflicting file names as any my-favorite-server.py process will likely have the same low, single or double digit PID in each of the multiple containers. As a result, the different processes will attempt to mmap the same file in the shared multiprocess metric directory, and various scary errors that aren't always easy to understand the root cause will occur.

It would be beneficial if something in addition to the PID was used to help ensure uniqueness across the files in the PROMETHEUS_MULTIPROC_DIR, or if a string could be passed in at initialization time to be embedded in the file names, such as a container or pod name, or another value determined by the application.

[csmarchbanks]

Ideally the multiproc directory would not be shared across pods/containers. Likely the best solution would be possible to set PROMETHEUS_MULTIPROC_DIR to contain the pod name or container name in the path, this is certainly possible with Kubernetes.

Since the entire directory is supposed to be wiped before application startup lots of issues can crop up if the directory is shared across applications.

[mkingsbury]

I have no issue wiping the directory, and yes, I would agree each distinctly different application should have its own directory. In our case, its one application running multiple identical pods behind a load balancer. We'd like to be able to report the metrics on the entire application, not 6 individual pods. With a load balancer in front, we don't know which pod we might be reaching when we query for the metrics.

[csmarchbanks]

Ahh, that is an interesting use case that is not supported today. Multiproc is designed to be run by one instance, but you want to aggregate across multiple instances. Typically this is handled by collecting data from each instance separately and aggregating in Prometheus, is the amount of data too high to do that? It would also give you visibility if a single pod is having issues.