Capture tracing when a specific header is passed in an HTTP request

[LeeHester]

We currently trace at a low sample rate (1%) in our web app. Sometimes, a trace would be really helpful in tracking down a problem for a specific user. I'd like to be able to add a special HTTP header (for example, "FORCE_DATADOG_TRACE" or something) to an HTTP request, have my rails app notice that, and trace that specific request, so that I can use the trace for debugging purposes. I tried doing this with a Rails middleware, like this:

(in config/application.rb)

  ...

  # This middleware checks for an HTTP header named "FORCE_DATADOG_TRACE", and
  # tells DataDog tracing to keep the trace for any request with this present.
  class ForceDataDogTraceMiddleware
    def initialize(app)
      @app = app
    end

    def call(env)
      if env["HTTP_FORCE_DATADOG_TRACE"]
        # Keeps the active trace
        Datadog::Tracing.keep!
      end
      @app.call(env)
    end
  end

  ...

  class Application < Rails::Application
    ...
    config.middleware.use ForceDataDogTraceMiddleware
    ...
  end

I debugged this locally and I see the Datadog::Tracing.keep! line of code is being executed when that header is passed, but I'm not getting those "forced" traces to show up in the DataDog UI. I'm not sure if this is because there is no active trace at the time, or if I'm just going down the wrong path here. Has anyone else tried to accomplish something similar?

[markedmondson]

We're doing something similar but closer to the controller...

    around_action do |controller, action|
      if params[:force_trace] == "true"
        Datadog::Tracing.trace("#{controller.class}##{params[:action]}") do |_span, trace|
          trace.keep!
          action.call
        end
      else
        action.call
      end
    end

This works on all except one application route (I ended up here because I'm trying to figure out why that one controller doesn't keep its traces!)

Hope this helps.

[LeeHester]

Mark, that is super helpful, thanks for sharing!

[marcotc]

👋, this is they're likely happening because when Datadog::Tracing.keep! is called, there's still no trace active: https://github.com/DataDog/dd-trace-rb/blob/7de53929e489ff6025855eb263bf79d82be48f43/lib/datadog/tracing.rb#L85

In your original example, you can guarantee that the trace is created before choosing a sampling decision (keep!) like so:

def call(env)
  if env["HTTP_FORCE_DATADOG_TRACE"]
    # Keeps the active trace
    Tracing.continue_trace!(nil) do
      Datadog::Tracing.keep!
      @app.call(env)
    end
  else
    @app.call(env)
  end
end

This is the cleanest way in my opinion, and it doesn't require you to create a dummy span (Datadog::Tracing.trace("#{controller.class}##{params[:action]}")).

Let me know if this works!

[ivoanjo]

Sounds like this issue was successfully fixed :)

I'm going to go ahead and close this one.