Filter span ingestion based on classes

[aovertus]

Opening an issue as requested here: https://datadoghq.dev/dd-trace-rb/#application-side-sampling

Current behaviour All spans are ingested

Expected behaviour Being able to filter which span is ingested based on different parameters (in our case, class name)

Steps to reproduce Trigger different controllers/jobs, see all traces being reported

How does ddtrace help you?

Will be testing this solution:

class CustomerSampler < Datadog::Tracing::Sampling::Sampler
  def sample?(trace)
    # Check if any span in the trace is from the TestClass class
    if trace.any? { |span| span.name == "TestClass" }
      rand < 0.5      
    else
      # Sample all other traces
      true
    end
  end
end

Datadog.configure do |c|
  c.tracing.sampling.default_rate = 1.0
  c.tracing.configure do |t|
    t.sampler = CustomerSampler.new
  end

  c.tracing.instrument :rails
  c.tracing.instrument :mysql2
  c.tracing.instrument :sidekiq, tag_args: true
  c.tracing.instrument :active_job
  c.tracing.instrument :graphql
end

Environment

  Datadog.configure do |c|
    c.tracing.sampling.default_rate = 1.0

    c.tracing.instrument :rails
    c.tracing.instrument :mysql2
    c.tracing.instrument :sidekiq, tag_args: true
    c.tracing.instrument :active_job
    c.tracing.instrument :graphql
  end

[marcotc]

👋 @aovertus, your example is pretty much a working configuration for class based sampling.

The only change you have to make is implement sample! instead of sample?:

class ClassSampler
  def sample!(trace)
    # Check if any span in the trace is from the TestClass class
    trace.sampled = if trace.any? { |span| insert_class_metcher_here(span) }
                      rand < 0.5
                    else
                      # Sample all other traces
                      true
                    end
  end
end

Datadog.configure do |c|
  c.tracer.sampler = ClassSampler.new
end

Also, what class would you like to filter by? Spans don't have an intrinsic concept of Ruby Class. Is it the class where the span was created? Or the resource class for the span (e.g. Sidekiq worker class name)? These can normally be filtered by matching against span.resource.

[aovertus]

Hi @marcotc

We use resource filtering to completely ignore resource such as healthcheck

        {
          "name": "DD_APM_IGNORE_RESOURCES",
          "value": "HealthChecksController#show$"
        }

In other use cases, we have sidekiq jobs which generate millions of traces (eg Notifications::SendJob). For these, we want to keep be able to monitor a sample (eg: 5%) for performance review.

[marcotc]

I see, something like this maybe is what you are looking for:

class ClassSampler
  def sample!(trace)
    # Check if any span in the trace is from the TestClass class
    trace.sampled = if trace.any? { |span| span.resource == ENV['DD_APM_IGNORE_RESOURCES'] }
                      rand < 0.5
                    else
                      # Sample all other traces
                      true
                    end
  end
end

Datadog.configure do |c|
  c.tracer.sampler = ClassSampler.new
end

[abennett]

How are you able to call trace.any? if it's not Enumerable, or am I missing something?