instana
[Bug]: Silently fails OTEL trace propagation
Problem Description
The standard open telemetry documentation includes the following snippet for trace propagation:
from opentelemetry import context as opentelemetry_context
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
carrier = {}
TraceContextTextMapPropagator().inject(carrier)
# Send to different process ...
ctx = TraceContextTextMapPropagator().extract(carrier=carrier)
opentelemetry_context.attach(ctx)
However, this silently fails with Instana. The traces are not connected and there is no warning emitted.
I tried using the Instana TextPropagator like the following:
from instana.propagators.text_propagator import TextPropagator
carrier = {}
TextPropagator().inject(current_span.get_span_context(), carrier)
# Send to different processs
ctx = TextPropagator().extract(carrier=carrier)
opentelemetry_context.attach(ctx)
But ran into errors around ctx being a SpanContext and not a normal Context.
The only way I was able to resolve this issue was to do something like the following where I use thread local variables to ensure I call start_span with the SpanContext
import threading
from instana.propagators.text_propagator import TextPropagator
CURRENT_INSTANA_SPAN_CONTEXT = threading.local()
carrier = {}
TextPropagator().inject(current_span.get_span_context(), carrier)
# Send to different processs
ctx = TextPropagator().extract(carrier=carrier)
CURRENT_INSTANA_SPAN_CONTEXT.context = ctx
# Later in code:
with tracer.start_span("span_name", span_context=CURRENT_INSTANA_SPAN_CONTEXT.context
I would like to be able to use standard OTEL trace propagation methods with Instana or have similar functionality
Minimal, Complete, Verifiable, Example
No response
Python Version
3.11
Python Modules
N/A
Python Environment
N/A
Hi @HonakerM,
ctx = TextPropagator().extract(carrier=carrier)
opentelemetry_context.attach(ctx)
This wouldn't work because as you've already mentioned extract() returns an Instana SpanContext object and not an OTel Context object.
[!NOTE] This is how we extract the
Contextobject from a span.
You can directly use the SpanContext for context propagation. Please note that if you want to create a new span and set it as the current span in this tracer’s context, you can use start_as_current_span() instead.
span_context = TextPropagator().extract(carrier=carrier)
with tracer.start_as_current_span("span_name", span_context=span_context):
...
Please do let me know if this serves your purpose or we can discuss further, thanks!
P.S. Python tracer does use OTel's API as a base but doesn't work together with OTel.
@GSVarsha okay it would be nice to have a way to propagate the contexts without having to pass it into the start of start_span(). The issue is that the code doing the propagation is not always the one starting new spans.
P.S. Python tracer does use OTel's API as a base but doesn't work together with OTel since we have our own tracer spec from Instana to follow.
Why was this the case and is there any documentation around this custom tracer spec?
Hello @HonakerM, we have an alternate way for context propagation (using SpanContext) unlike OTel. So currently, there's no direct way to extract the Context object.
The issue is that the code doing the propagation is not always the one starting new spans.
Would be interesting to know what you're trying to achieve here, thanks!
@GSVarsha does the Instana team have any documentation on why choosing a different context propagation system?
The issue is that the code doing the propagation is not always the one starting new spans.
Our distributed processes communicates via queues and DBs and don't directly interact with each other. We have to store the system info in the DB and then send the context with the request over the queue. The receiving process doesn't know what span to start until it's retrieved data from the DB but it already has the context so there's a bit of a gap