prometheus
Why is there no examplar support on multiprocss mode?
I understand why this is the case for e.g. Gauges in min/max/sum mode, but not for all/liveall, where we can differentiate by a pid label (or similar), and similarly for histograms and counters with a pid-like label. Does something prevent supporting exemplars in those caess?
The simple answer is that it hasn't been implemented yet. There will be a bit of a technical challenge in order to store and update the exemplars in the the appropriate db files, but it should be doable. I haven't had the time to add support yet, but I would be happy to review a PR if someone can contribute sooner.
Is there a quick overview of what someone's plan for adding support would be?
A couple options that I have considered:
Very basic support might be possible by keeping an exemplar in memory for each process, then whichever process happened to handle a request could provide an exemplar. I would want to check to see if Prometheus would handle constantly switching exemplars before merging a solution like that.
For full support it will be necessary to either modify https://github.com/prometheus/client_python/blob/master/prometheus_client/mmap_dict.py to add support for adding exemplars to the .db files created for each process, or else create a second batch of .db files that are exemplar specific. My intuition is that an exemplar specific file might be safer and more performant for metrics that do not have exemplars.
According to the OpenMetrics Spec the combined length of labelnames and labelvalues on an exemplar cannot exceed 128 UTF-8 characters which could be used to create entries in a .db file similar to the 8 byte max size for a value in the current .db files. Exemplars are much larger than values, which is why we may need to have a separate file for exemplars to avoid allocating a bunch of extra space for metrics without exemplars.