proposal: allow setting sampling rate per trace

[gbbr]

This proposal is to allow setting a custom sampling rate per span. It suggests introducing a new StartSpanOption that sets the sample rate:

// WithSamplingRate allows setting a custom sampling rate for a trace on a root span. If the
// option is used on a child, it will be ignored.
func WithSamplingRate(rate float64) StartSpanOption {
    return Tag(sampleRateMetricKey, rate)
}

This would then set the standard sampling rate metric on the root span, and allow it to get special treatment inside the internal sampling function. For example: if this tag is found, we can use sampledByRate and return.

This will allow for a span to be started with a custom sampling rate as such:

tracer.StartSpan("root", tracer.WithSamplingRate(0.4))

[kevinconaway]

This would then set the standard sampling rate metric on the root span

This works when the caller controls the start of the span. However, when the span is started upstream from userspace code, for example in a grpc server interceptor, the user would not be able to use this option.

That particular case could be difficult for users to reason about. Does that make sense ?

[gbbr]

I understand. I guess the natural question to ask is: why did you exclude the use of sampling priority? For example, you could do:

span.SetTag(ext.ManualDrop, true)

Or, if you're using a version that is older than 1.13.0:

span.SetTag(ext.SamplingPriority, ext.PriorityUserReject)

[kevinconaway]

why did you exclude the use of sampling priority?

For some operations in our application, we want all of the traces to be sent (as in, the default sample rate of 1). One particular high volume operation was overflowing the tracing buffer and we started dropping traces.

My thought was that we could use a sampling rate to only keep N % of those request, not drop them entirely or unpredictably.

The sampling rate proposal here would work for our use case perfectly but I'd just like to make sure that it fits in nicely with others as well

[knusbaum]

Closing this as rejected.

We have newer sampling mechanisms now which should allow users to sample different resources/operations at different rates.