How to update the trace_id of the current span?

[rsb177]

Tracer Version(s)

2.20.5

Python Version(s)

Python 3.12

Pip Version(s)

pip 24.0

Bug Report

I don't think this is really a bug, but I'm trying to set the trace_id of the current span (my function is wrapped by the tracer.wrap decorator). When I try it with tracer.current_span().trace_id = trace_id I don't get any errors, but it doesn't report in to datadog anymore. The only way I've been able to successfully set the trace_id is by not using the decorator and instead using the context manager inside the function itself (setting the trace_id before the context manager). Is there a way to do this with the decorator?

Reproduction Code

No response

Error Logs

No response

Libraries in Use

No response

Operating System

No response

[wantsui]

Hey @rsb177 ! Can you provide a code snippet showing these two scenarios?

With this being one scenario:

The only way I've been able to successfully set the trace_id is by not using the decorator and instead using the context manager inside the function itself (setting the trace_id before the context manager). Is there a way to do this with the decorator?

And this being another:

When I try it with tracer.current_span().trace_id = trace_id I don't get any errors, but it doesn't report in to datadog anymore.

Additionally what's the use case that you have where you need to override the default trace id? I'm also wondering if there's a different approach for it vs the override.

Thanks in advance!

[rsb177]

working:

def consumer_job(event):
    context = Context(trace_id=trace_id)
    tracer.context_provider.activate(context)

    with tracer.trace("consumer_job", service=settings.DD_SERVICE):
        pass

not working:

@tracer.wrap("consumer_job", service=settings.DD_SERVICE
def consumer_job(event):
    trace_id = get_trace_id(event)
    tracer.get_current_span().trace_id = trace_id

In this case I'm processing events from Kafka that have their own trace id. I'm trying to set the trace_id to link across applications.

[wantsui]

Oh! Thanks for this context, I'll review with my team @rsb177 !

[wantsui]

Hi @rsb177 , I'm seeing a few issues for this thread to tackle, but I just want to ask a clarifying question - are you using confluent-kafka for these kafka calls?

If so, I actually see a setting in our Confluent Kafka code for DD_KAFKA_PROPAGATION_ENABLED, which is disabled by default. When enabled it allows the kafka producer and consumer to inject and extract context so they should connect in the same trace. https://github.com/DataDog/dd-trace-py/blob/73764dc6414f1ce7e9f4e176e1273be675b3d632/ddtrace/contrib/internal/kafka/patch.py#L47

Otherwise, I noticed you are creating your own consumer job spans, so I was wondering this is because you're using a Kafka library that we don't support? If so, which one?

(It doesn't change the fact that I still need to look at the discrepancy between the two types of custom spans - I'm still researching that!)

[rsb177]

We're not using confluent-kafka, we're using aiokafka (we're actually using our own wrapper around aiokafka) which I don't believe is supported.

[wantsui]

Got it, thanks I'll just focus on the difference in trace ids between the two methods (I was worried we had a bug in confluent-kafka somewhere!)

[wantsui]

I don't have the definition for get_current_span, but on dd-trace-py 2.21.4, I was able to override the trace id for the "not working" example:

Code:

from ddtrace import tracer
from ddtrace._trace.context import Context

def consume(trace_id):
    context = Context(trace_id=trace_id)
    tracer.context_provider.activate(context)
    with tracer.trace("consume_span"):
        print(f"I am a custom span created with tracer.trace and my trace id is {tracer.current_span().trace_id}")
        pass

@tracer.wrap("consume_span")
def consume_wrap(trace_id):
    print(f"I am a custom span created with tracer.wrap and my trace id is currently {tracer.current_span().trace_id}")
    tracer.current_span().trace_id = trace_id
    print(f"I am a custom span created with tracer.wrap and my trace id is now {tracer.current_span().trace_id}")


consume(2)
consume_wrap(3)

requirements.txt

ddtrace==2.21.4

Result:

I am a custom span created with tracer.trace and my trace id is 2
I am a custom span created with tracer.wrap and my trace id is currently 138225844890171679213525983470252511256
I am a custom span created with tracer.wrap and my trace id is now 3

Is there anything my repro attempt is doing that your setup doesn't have? (For example, what does get_current_span do?) In our docs we expect current_span to access the current span, as seen here: https://docs.datadoghq.com/tracing/trace_collection/custom_instrumentation/python/dd-api/?tab=decorator#accessing-active-spans .

[rsb177]

Appologies, i meant current_span not get_current_span. For the not working example I was able to change the trace id and it doesn't error. However it stops reporting to datadog then. Nothing shows up in the logs either.

[wantsui]

Thanks for the clarification! I am able to reproduce this behavior now. It looks like there's an issue when we try to finish the span where the trace id gets manually overridden outside of the context provider, since in debug mode, I see the error finished span not connected to a trace being thrown.

I'll ask my team what they think and get back to you!

[wantsui]

As an update after syncing with my team, we don't have short term plans to allow editing the trace id outside of tracer.context_provider.activate since it can cause broken tracing behavior. Meaning tracer.current_span().trace_id = trace_id won't be a support edge case for now.

I'll mark this as a feature request instead since I agree it's not a bug.

In terms of a workaround, is there a reason you have to use the decorator? I'm wondering if you can rely on your working example instead?

Thanks!

[github-actions[bot]]

This issue has been automatically closed after a period of inactivity. If it's a feature request, it has been added to the maintainers' internal backlog and will be included in an upcoming round of feature prioritization. Please comment or reopen if you think this issue was closed in error.