opentelemetry-collector-contrib icon indicating copy to clipboard operation
opentelemetry-collector-contrib copied to clipboard

Probabilistic sampler: emit metrics for number of spans sampled

Open shinkle-procore opened this issue 7 months ago • 7 comments

Component(s)

processor/probabilisticsampler

Is your feature request related to a problem? Please describe.

We are trying to build SLOs (Service Level Objectives) around our telemetry pipeline. One of the indicators we use is the ratio of "spans received by receiver" to "spans sent by the exporter" expecting that ratio to stay close to 1 (otherwise it means we're dropping spans during processing).

Now that we have probabilistic sampling set up (at 10%), we're finding that ratio is around 0.1 instead of 1, which makes sense. However, just adding a factor of 10 to our SLO measurement to compensate is hacky. We would like to factor something like "otelcol_processor_probabilistic_sampler_count_spans_sampled" into our calculation, but right now we only have access to "otelcol_processor_probabilistic_sampler_count_traces_sampled"

The telemetry in the core collector library seems to revolve around "spans" while the sampler metrics revolve around "traces", so it's difficult to compare the two in calculations.

Describe the solution you'd like

Emit a metric of "otelcol_processor_probabilistic_sampler_count_spans_sampled" from the probabilistic sampler processor.

Describe alternatives you've considered

No response

Additional context

No response

shinkle-procore avatar Jul 26 '24 18:07 shinkle-procore