Spans not recorded by default if context is propagated

[mzealey]

Describe your environment

OS: Ubuntu 2204 Python version: 3.10.12 Package version: 1.3.0/0.51b0

What happened?

I originally raised this issue as https://github.com/open-telemetry/opentelemetry-python-contrib/issues/3267 but now realise it seems to be a core bug rather than in a particular instrumentor.

The root of this issue appears to be that the propagator in https://github.com/open-telemetry/opentelemetry-python/blob/main/opentelemetry-api/src/opentelemetry/trace/propagation/tracecontext.py#L85-L87 returns a NonRecordingSpan if traceparent is set, however if it is not set a default _Span is created. https://github.com/open-telemetry/opentelemetry-python/blob/main/opentelemetry-sdk/src/opentelemetry/sdk/trace/init.py#L1160 then converts this into an _Span iff sampler is specified and returns that we should sample this trace.

The net result is some very confusing (and I think undocumented) behaviour: By default the root span is recording no matter whether a sampler is used or not. But if a valid traceparent header is seen it defaults to not recording unless there is a sampler set and it returns True (eg OTEL_TRACES_SAMPLER=always_on)

Steps to Reproduce

from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

resource = Resource.create()
trace.set_tracer_provider(TracerProvider(resource=resource))
span_processor = BatchSpanProcessor(ConsoleSpanExporter())
trace.get_tracer_provider().add_span_processor(span_processor)

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)


@app.get("/")
def root():
    return {"message": "Hello World"}

Run as fastapi dev v.py

Expected Result

Test with:

curl http://127.0.0.1:8000
curl http://127.0.0.1:8000 -H "traceparent: 00-94df63b03874c83da3cc8e207789df94-17fb4659f79f7ca2-00"

I would expect this to log to 2 traces to the console.

Actual Result

Only the first trace is logged, the one with traceparent is dropped.

Run again with OTEL_TRACES_SAMPLER=always_on fastapi dev v.py and repeat the curls and both traces appear on the console.

Additional context

https://github.com/open-telemetry/opentelemetry-python-contrib/issues/3267

Would you like to implement a fix?

None

[mzealey]

I should add this also happens with the flask implementation hence raising it centrally here.

[pszpetkowski]

Your traceparent value is 00-94df63b03874c83da3cc8e207789df94-17fb4659f79f7ca2-00 so therefore through trace flags with value 00 you've told otel that it shouldn't sample this trace, that's why it's a NonRecordingSpan.

https://www.w3.org/TR/trace-context/#sampled-flag