Inconsistent sentry-trace and baggage values on incoming trace

[sl0thentr0py]

Version

incoming trace version - 2.11.0, outgoing trace version - unknown

Steps to Reproduce

https://sentry.sentry.io/api/0/projects/sentry/sentry/events/6122ec07d3694b7ba1aa88db962afde5/json/

• trace_ids on the two headers are different
• sentry-trace is sampled but the baggage is not

sentry-trace:  ac9fb4acad4a4f3d90ea2660a9d060b9-aa7d2d9bb68db6d5-1
baggage:       sentry-trace_id=85f29184b20d47c28dde78483288c3d7,sentry-sample_rate=0.0,sentry-sampled=false,sentry-environment=stage,sentry-public_key=[REDACTED]

the request comes from 127.0.0.1 which are our internal proxies. they usually relay the client's IP unless the request actually originates internally. Combined with the fact that we're dealing with a silo RPC, the request that triggered this transactions may come from an internal service. However, this may in turn have been triggered by an external request. That I/we don't know.

[scott-schibli]

@sl0thentr0py I am seeing this as well in my project. Im curious, are you seeing these inconsistencies and still able to effectively use distributed tracing across systems?

I am running into an issue where each new request to my BE from my FE / browser is causing these two methods below to return a very similar output as you see above:

sentry_trace = sentry_sdk.get_traceparent()
sentry_baggage = sentry_sdk.get_baggage()
// then print in a dict i get something like this: 

{
'sentry_trace': '3311a76eb75844509d1e4406e3efc075-b9d29f60220bc5f0-1', 
'baggage': 'sentry-trace_id=c867096f09a44c67892077b37d7cfd15,sentry-environment=development,sentry-release=777d725ed6de3660deaf1996ea88ceb847d21cfe,sentry-public_key=18a4b04ee1c04110abcb6302a134e3fa,sentry-transaction=/,sentry-sample_rate=1.0,sentry-sampled=true'
}

In my situation I have a my FE setup with sentry/browser sdk, and my backend is with the django sdk. My app is SSR, so this is the flow I have:

• I add the meta tags to my html template with the sentry-trace and baggage values (This works as intended, when i throw a FE error, the trace starts with my BE / initial server request, and then shows the FE transactions before error).
• I have allowed all targets with tracePropagationTargets as I am just testing as of now, which adds the sentry-trace and baggage headers (this also works as expected).
• So, at this point the page has loaded, when I finally make a request to my api / BE to an endpoint that throws and error sentry starts a new trace even though the sentry-trace and baggage headers are in the request header.

The interesting thing, is when I log the get_traceparent and get_baggage values I see something very similar to you.

// headers on request: 
sentry-trace: c867096f09a44c67892077b37d7cfd15-...-1
baggage: c867096f09a44c67892077b37d7cfd15-...-1, sentry-trace_id=c867096f09a44c67892077b37d7cfd15,sentry-environment=development ... (the rest)

// 
sentry_sdk.get_traceparent(): 3311a76eb75844509d1e4406e3efc075-...-1
sentry_sdk.get_baggage(): sentry-trace_id=c867096f09a44c67892077b37d7cfd15, ... (the rest)

So you can see that the baggage sentry-trace_id is actually the correct one from the header that is being passed from the FE, but my django sentry setup is not using that trace and is rather starting a new one.

I may open my own issue, but I saw a similarity from your post to what I am seeing and wanted to see if this give you an idea of what could be wrong, and also is the distributed tracing is still working for you even though you have this inconsistency?

[sl0thentr0py]

thx for the report @scott-schibli, I haven't gotten time to debug this at the moment but I'll keep you posted when I pick this up. There's definitely some sort of case we're not handling properly.

[scott-schibli]

@sl0thentr0py Do you have any inclination to why distributed tracing isn't continuing the trace with the django sdk even when the sentry-trace and baggage headers are present? Documentation from the sentry/browser and django sdk says that distributed tracing for django system should work out of the box when these headers present. In the meantime I will try to manually continue the trace in our middleware when the headers are present.

[antonpirker]

Hey @scott-schibli !

I have now tested this with a simple Django application that has a Vue JS application in the index.html template: https://github.com/antonpirker/testing-sentry/tree/main/test-django-tracing-js

I have a index view that renders the trace information to HTML meta tags: https://github.com/antonpirker/testing-sentry/blob/main/test-django-tracing-js/mysite/polls/views.py

In the index page a Vue JS app is loaded that makes a fetch request to the API endpoint. In the API endpoint in Django there is an error.

I have the correct trace information in the sentry-trace and baggage fields in the HTML meta tags, the outgoing http request headers, and the /api endpoint. So everything works as excepted.

What you can try further to debug your case:

• Check the values of sentry-trace and baggage in the server rendered HTML meta tags are.
• Check the the HTTP request headers from the outgoing request from your frontend are the same.
• Check if the incoming request headers in your API endpoint are also the same.

My hunch is now that your proxy (or something else) is caching some headers or somehow changes them...

[antonpirker]

Because this issue is not reproducible, I will close this issue.

@scott-schibli If you have more information of your system and can point to where the sentry-trace or baggage fields are changed, then please create a new issue. Thanks!