prometheus-engine icon indicating copy to clipboard operation
prometheus-engine copied to clipboard

[Do not Merge] Support exporting exemplars to GCM

Open realschwa opened this issue 3 years ago • 0 comments

This PR adds support for exporting prometheus exemplars into Google Cloud Monitoring.

I have added a parameter of type []record.RefExemplar called exemplarBatch to the Export function in export.go. The Export function then constructs a map from a chunks.HeadSeriesRef to the exemplar so that we can quickly grab the correct exemplar when processing a sample from a particular series. (NOTE: I think these will never be out of order anyway, e.g. if we are looking at the head of the samples array and the head of the exemplars array they either match or the head of the series array has no exemplar. In other words, the relative ordering of the samples and exemplars are both the same with respect to the series to which they belong. If this is true, then we do not need to construct this map.)

The exporter passes this into the buildDistribution function and properly stores the exemplar associated with each sample in its associated distribution. When we convert the LabelSet into attachments, if we see the following three fields in the LabelSet:

  1. trace_id
  2. span_id
  3. project_id Then we will use it to create a SpanContext and put the remaining label/value pairs into the DroppedLabels attachment. If all three of these labels are not present, then all label/value pairs -- including any of the above -- will go into the DroppedLabels attachment.

There is also a new ExporterOpt called InferSpanProjectID. If this is set to true then users will not have to supply a project_id into an exemplar. We will instead use the one that we are writing metrics to.

Note: We should only have one exemplar per series in any call to export. Also note: This requires a change in calling export in our prometheus fork.

realschwa avatar Aug 23 '22 18:08 realschwa