traceparent support

[spunkedy]

when the traceparent header is passed on the requests to phoenix, it is currently not supported to be able to pull in the parent span so that we can support having distributed tracing until the full otel support is implemented

[spunkedy]

[spunkedy]

      {:sentry, git: "https://github.com/spunkedy/sentry-elixir", branch: "feature-enable-traceparent"},

has been tested with passing the headers

[solnic]

Thank you for the PR! 💜 We actually need to introduce an opentelemetry propagator as we have an architectural design for this functionality that is followed by other SDKs and we need the Elixir SDK to follow it as well 🙂

If this is something you'd be interested in doing - you could port what we do in the Ruby SDK: https://github.com/getsentry/sentry-ruby/blob/master/sentry-opentelemetry/lib/sentry/opentelemetry/propagator.rb

[venkatd]

There's an older implementation, but I tested it and it doesn't work with the latest packages: https://github.com/scripbox/opentelemetry_sentry/blob/v0.1.1/lib/opentelemetry_sentry/propagator.ex

I believe the main issue is that baggage is not being propagated

[venkatd]

I wanted to mention that we have a working prototype of distributed tracing. It requires a custom propagator and some tweaks to the span processor to support forcing a transaction w/ http requests.

I don't have time at work to take this to completion but sharing in case it's helpful to anyone.

We use this custom propagator.

defmodule Ex.Sentry.OpenTelemetryPropagator do
  @behaviour :otel_propagator_text_map

  require Record
  require OpenTelemetry.Tracer, as: Tracer

  @fields Record.extract(:span_ctx, from_lib: "opentelemetry_api/include/opentelemetry.hrl")
  Record.defrecordp(:span_ctx, @fields)

  @sentry_trace_key "sentry-trace"
  @sentry_baggage_key "sentry-baggage"
  @sentry_trace_ctx_key :"sentry-trace"
  @sentry_baggage_ctx_key :"sentry-baggage"

  @impl true
  def fields(_opts),
    do: [@sentry_trace_key, @sentry_baggage_key]

  @impl true
  def inject(ctx, carrier, setter, _opts) do
    case Tracer.current_span_ctx(ctx) do
      span_ctx(trace_id: tid, span_id: sid, trace_flags: flags) when tid != 0 and sid != 0 ->
        setter.(@sentry_trace_key, encode_sentry_trace_id({tid, sid, flags}), carrier)

      _ ->
        carrier
    end
  end

  @impl true
  def extract(ctx, carrier, _keys_fun, getter, _opts) do
    case getter.(@sentry_trace_key, carrier) do
      :undefined ->
        ctx

      header when is_binary(header) ->
        {trace_hex, span_hex, sampled} = decode_sentry_trace_id(header)

        ctx =
          ctx
          |> :otel_ctx.set_value(@sentry_trace_ctx_key, {trace_hex, span_hex, sampled})
          |> :otel_ctx.set_value(
            @sentry_baggage_ctx_key,
            baggage(getter.(@sentry_baggage_key, carrier))
          )

        # Create a remote, sampled parent span in the OTEL context.
        # We will set to "always sample" because Sentry will decide real sampling
        remote_ctx =
          :otel_tracer.from_remote_span(hex_to_int(trace_hex), hex_to_int(span_hex), 1)

        Tracer.set_current_span(ctx, remote_ctx)
    end
  end

  defp encode_sentry_trace_id({trace_id_int, span_id_int, sampled}) do
    sampled = if sampled, do: "1", else: "0"

    int_to_hex(trace_id_int, 16) <> "-" <> int_to_hex(span_id_int, 8) <> "-" <> sampled
  end

  defp decode_sentry_trace_id(
         <<trace_hex::binary-size(32), "-", span_hex::binary-size(16), "-",
           sampled::binary-size(1)>>
       ),
       do: {trace_hex, span_hex, sampled == "1"}

  defp decode_sentry_trace_id(<<trace_hex::binary-size(32), "-", span_hex::binary-size(16)>>),
    do: {trace_hex, span_hex, false}

  defp baggage(:undefined), do: nil
  defp baggage(""), do: nil
  defp baggage(bin) when is_binary(bin), do: bin

  defp hex_to_int(hex) do
    hex
    |> Base.decode16!(case: :mixed)
    |> :binary.decode_unsigned()
  end

  defp int_to_hex(value, num_bytes) do
    value
    |> :binary.encode_unsigned()
    |> bin_pad_left(num_bytes)
    |> Base.encode16(case: :lower)
  end

  defp bin_pad_left(bin, total_bytes) do
    missing = total_bytes - byte_size(bin)
    if missing > 0, do: :binary.copy(<<0>>, missing) <> bin, else: bin
  end
end

There are some details I'm not certain of but it is working for us.

For the span processor, we do need to ensure a new transaction is created. If we don't do this (from what I tested) Sentry will not show a a distributed trace.

I think a simpler check for an http request in the span processor would be as follows. I don't think it is necessary to test all of the fields.

  defp is_http_server_request_span?(%{kind: kind, attributes: attributes}) do
    kind == :server and Map.has_key?(attributes, to_string(HTTPAttributes.http_request_method()))
  end

Then the check can be:

    is_transaction_root =
      span_record.parent_span_id == nil or
        is_http_server_request_span?(span_record)

You can see it in action here:

[solnic]

@venkatd thanks man, very helpful!

[kubosuke]

no updates for this topic?

[solnic]

@kubosuke it's a WIP, I'll be opening a PR before the end of the week, got it working already just need to clean up the implementation :)