Passing a parent context to a new span

[ggpasqualino]

trafficstars

Hi there, I have noticed that in 0.5.0 neither start_span nor with_span functions in the module otel_traceraccept the parent option anymore. The start_span function is accepting now a context parameter that seems to be used for that now, but the with_span doesn't have a corresponding one. And in the Elixir API, none of them accept this now.

I can take a look at adding the new functions. What do you think?

[tsloughter]

@ggpasqualino if none of the Elixir API functions accept a context then this is a bug.

Argh, and you are right about with_span.

Sorry about that, these are quick fixes I can get done today and release 0.5.1

[ggpasqualino]

@tsloughter awesome, thank you!

[tsloughter]

Oh.. I now see the bigger issue. The Elixir Tracer module had been only including macros that act on the current context with the assumption that anything that the user required passing in either a context variable or a tracer they would use otel_tracer.

I think that idea doesn't hold now that parent can only be passed via a Context. Still should be a simple enough change, I think.

[pallix]

I want to reuse the trace_id of another service, to correlate traces from different services. I manage to write this proof of concept:

require Record
@fields Record.extract(:span_ctx, from_lib: "opentelemetry_api/include/opentelemetry.hrl")
Record.defrecordp(:span_ctx, @fields)

def hello do
    {trace_id, ""} = Integer.parse("0000000000123457", 16)
    span_id = :opentelemetry.generate_span_id()
    parent_span_ctx = span_ctx(trace_id: trace_id, span_id: span_id, tracestate: [])
    ctx = %{{:otel_tracer, :span_ctx} => parent_span_ctx}

    Tracer.with_span ctx, "hello4.1" do
      Process.sleep(200)
    end
end

It was hard to figure it out. Is there a simple way to achieve that?

[tsloughter]

@pallix yes, you should reuse the context or span context, not simply the trace id. Is there a reason you are only reusing the trace id?

[pallix]

yes, you should reuse the context or span context, not simply the trace id. Is there a reason you are only reusing the trace id?

Not really, I'm unfamiliar with the API and trying to understand it. If I want to correlate two spans from two different services, do I need to specify both trace_id and span_id or is specifying span_id enough?

[tsloughter]

What do you mean by "two different services"? Do you have one service talking to another over like HTTP?

[pallix]

Two different services taking with distributed erlang.

[tsloughter]

@pallix ah, then you can either create a new span and pass it as a variable or pass the context and then attach it in the other process.

I started on docs for this but need to finish them after adding a couple configuration options, but I created https://gist.github.com/tsloughter/ddd8653dc245b88c9d612fc5d85fe4e2 when working with someone else who was using Elixir and wanting to link to a span from another process.

It'd be basically the same thing as the code in link but instead of attaching in Task.async you'd pass ctx in a message and then attach it where you receive it.

[pallix]

Thanks for taking time to answer me and making this example!

I have some uuid coming in a field of the data structure received by the service and I was thinking about using it to create a top span id. This way I would not need to pass an extra argument to the services (they could derive it from the data content). It's not possible to create a span with a specific id yet, right? In the Rust opentelemetry api it's possible to do it (I'm looking into it also because we have a Rust NIF, it's called with_trace_id).

[brunoripa]

Hi, I have a very similar problem, I need to do the same but over 2 different apps (two elixir apps sharing messages over RabbitMQ). I serialize the span context with OpenTelemetry.Tracer.current_span_ctx() and generate something like:

{
   attachment_id: 33, 
   encoded_span_context: {
         is_recording: true, 
         is_remote: :undefined,
         is_valid: true, 
         span_id: 17678254447660265935,
         trace_id: 231436546593055454662563488462183936314,
         tracestate: :undefined
   }
}

The consumer receives this payload, and it should somehow reinflate the context; I have not really understood what's the procedure for this. In your gist example, the parent and parent_ctx objects are known, but not in mine, and the attach approach is not really verifiable, since I have one more problem, which is the following.

Also, and more importantly, I have seen that the trace_id coming from current_span_ctx() is not present in Zipkin: it has a specific "search by Trace ID" input field, but no traces are found that way, because they don't exist.

@tsloughter thanks for any help !

[bryannaegele]

@brunoripa you can just encode and decode headers to your rabbit message, similar to http requests. There's nothing official for rabbit yet as far as I know but it's on my list to tackle.

Here's how we do it. Note this is v0.4 code but you should get the idea.

Publisher

defp publish(payload, extra_headers, exchange) do
  attributes = attributes(payload)

  Tracer.with_span :"amqp:publish", %{kind: :PRODUCER, attributes: attributes} do
    data = payload |> Jason.encode!()
    headers = headers(extra_headers)

    # rest of your code
  end
end

defp headers(list) do
  list
  |> :ot_propagation.http_inject()
  |> Enum.map(fn
    {_, _, _} = tuple -> tuple
    {key, value} when is_list(value) -> {key, :longstr, :erlang.iolist_to_binary(value)}
  end)
end

Consumer

defp consume(
         %{connection: connection} = state,
         data,
         %{delivery_tag: tag, redelivered: redelivered, headers: msg_headers}
       ) do
  {trace_headers, headers} = parse_headers(msg_headers)

  trace_ctx = :ot_propagation.http_extract(trace_headers)

  Tracer.with_span :"amqp:message:consume", %{kind: :CONSUMER, parent: trace_ctx} do
    # your code
  end
end

defp parse_headers(headers) when is_list(headers) do
  headers
  |> Enum.reduce({[], []}, fn header, {trace_headers, headers} = acc ->
    case header do
      {k, _type, v} when k in ["traceparent", "tracestate"] ->
        {[{k, v} | trace_headers], headers}

      {k, _type, v} ->
        {trace_headers, [{k, v} | headers]}

      _ ->
        acc
    end
  end)
end

defp parse_headers(_), do: {[], []}

[tsloughter]

Yup, still want to use the text map propagator. And also might want to make a link in the consumer instead of making it a child.

[bryannaegele]

Most vendors don't support links yet afaik. I know Lightstep definitely doesn't. The link vs child is also pretty contextual to the use case. I haven't figured out quite how we'd distinguish that, but I reckon it's going to have to remain 100% contextual to the consumer.

[brunoripa]

Wow, guys, thanks for the quick response to you all <3 ... I will try ! @bryannaegele I am actually switching from jaeger to zipkin, for the records.

[brunoripa]

@bryannaegele can I ask you where does :ot_propagation come from ? I can only find :ot_propagator ...

[garthk]

They renamed to an otel_ prefix, so it's now :otel_propagator. Try this:

iex> require OpenTelemetry.Tracer, as: Tracer
OpenTelemetry.Tracer

iex> Tracer.with_span "foo", do: :otel_propagator.text_map_inject([])
[{"traceparent", "00-a0cf624b64a282ca388c8861dc590eb6-63a6afb55364cf47-01"}]

iex> Tracer.current_span_ctx()
:undefined

iex> :otel_propagator.text_map_extract([{"traceparent", "00-a0cf624b64a282ca388c8861dc590eb6-63a6afb55364cf47-01"}])               
:ok

iex>  Tracer.current_span_ctx()
{:span_ctx, 213753278424701560020113743190167195318, 7180619849211891527, 1,
 :undefined, :undefined, :undefined, false, :undefined}

[dvic]

@garthk Related to this, I stumbled upon a tweet from you where you talk about getting the context in a plug (started from a cowboy telemetry handler). I see the PID of the adapter (cowboy) process in the adapter field of Conn, but could you share how you actually shared the context from the cowboy process with the plug, using an ETS table or something else?

[dvic]

FYI I came up with this idea (https://github.com/opentelemetry-beam/opentelemetry_phoenix/issues/28#issuecomment-887421593):

Create the parent span inside a cowboy stream handler, and inject the context into the headers using :otel_propagator.text_map_inject?

Defining and using a Cowboy stream handler in Elixir:

defmodule DemoWeb.CowboyHandler do
  @behaviour :cowboy_stream

  # implement all opentelemtry logic in this module

  @impl true
  def init(stream_id, req, opts) do
    :cowboy_stream_h.init(stream_id, req, opts)
  end

  @impl true
  def info(stream_id, info, state) do
    :cowboy_stream_h.info(stream_id, info, state)
  end

  @impl true
  def data(stream_id, is_fin, data, state) do
    :cowboy_stream_h.data(stream_id, is_fin, data, state)
  end

  @impl true
  def terminate(stream_id, reason, state) do
    :cowboy_stream_h.terminate(stream_id, reason, state)
  end

  @impl true
  def early_error(stream_id, reason, partial_req, resp, opts) do
    :cowboy_stream_h.early_error(stream_id, reason, partial_req, resp, opts)
  end
end

and then in dev.exs

stream_handlers: [:cowboy_telemetry_h, DemoWeb.CowboyHandler]

[garthk]

I'm sorry, I don't have the code any more. I used telemetry and cowboy_telemetry rather than my own stream handler; good on you! Either way, the trick was to have the stream handling process store the span_ctx record or a traceparent derived from it in an ETS table—good guess!—and then have the connection process query it and set its span context. Good luck!

[tsloughter]

Closing this because the Elixir API with_span accepts a Context.