b3-propagation
b3-propagation copied to clipboard
openzipkin
How does a caller figure out the trace id of a request?
For debugging or (log) correlation reasons, a user might want to know what the trace id is of a request. Since X-B3 headers are request headers (not response headers), how does a user know which trace is related to their request?
There's a number of ways to accomplish this.
Specify the X-B3-TraceId and X-B3-SpanId headers
In B3, the recipient of X-B3-TraceId and X-B3-SpanId resumes that span, even if they immediately create a child. If you want to know the trace identifiers, specify them!
Keep in mind that unless X-B3-Sampled: 1, the trace might be dropped due to sampling. If you are just interested in controlling ids, leave out the X-B3-Sampled header and let the next hop decide. If you need the specific trace to be sampled, set X-B3-Flags: 1, indicating it is a debug span requested to pass all sampling (even collection tier).
For example, here's zipkin's browser extension controlling trace identifiers https://github.com/openzipkin/zipkin-browser-extension/blob/master/js/attachBeforeSendHeadersListener.js#L15
Add a binary annotation corresponding to a application-defined request id
While B3 doesn't affect the backtrace (response headers), applications often do. For example, they might return x-myapp-requestid: xxxxxxx. Zipkin has key-value search capabilities based on "binary annotations" which is a fancy name for tags. When a tracer starts a Span associated with a B3 initiated trace, it can add the application request id as a binary annotation in the root span. Then, a user can search by the request id they know.
I'd say X-B3-TraceId is all a user may want and it can be just added to response headers. At least in the implementations I deal with it would be trivial as it is a middleware in the request/response stack
We return X-B3-TraceId in response headers by default. Unfortunately there are some cases where a caller needs to know what the trace ID will be before it makes the call (i.e. in the case of fire & forget scenarios where the response cannot be inspected) - in those cases the caller sends in the B3 headers it wants to be used. So :+1: for @adriancole's description covering both cases.
I didn't mention my opinion on returning X-B3-TraceId.. something I should have. First, the audience for this project are hopefully less than 50 developers of tracing libraries, not end-users.
Long story short is that I think it is very worthwhile to put docs around what tracer authors should know about when thinking B3, I'd be careful to propagate B3 directly to end users as this limits our ability to change in the future.
X-B3-TraceId isn't a great user-level id. In fact, I suspect we will want to change to a new propagation model in the future. The more users directly interact with X-B3-TraceId the tougher it is to ever change to a new propagation model.
specifying ids on the way out should ideally be done via a tool like an instrumented curl or swagger thing. That way, it can change easily (as that tooling is centralized and has a changelog). Getting users used to B3 like they are Content-Type might solidify something late in the game. We may wish to change to a more conscious approach of id provisioning in the future. Ex one that more explicitly understands the nature of request ids vs trace ids, like envoy https://lyft.github.io/envoy/docs/intro/arch_overview/tracing.html cc @mattklein123
your thoughts?
ex this is a swagger decorator which allows power-users a way to initiate traces, using B3 without knowing how it works https://github.com/Yelp/swagger_zipkin by @prat0318
@adriancole I don't have a lot of context on this, but FYI @RomanDzhabarov did all of the tracing work in Envoy if you have any questions and some folks from LightStep might also be interested in this (@jmacd @djspoons @bensigelman).
Ah cool just pinged you on account of commit volume to envoy (also assumed some x-twitter folks might be implicated in envoy and may have known about B3)
On 16 Sep 2016 11:15, "Matt Klein" [email protected] wrote:
@adriancole https://github.com/adriancole I don't have a lot of context on this, but FYI @RomanDzhabarov https://github.com/RomanDzhabarov did all of the tracing work in Envoy if you have any questions and some folks from LightStep might also be interested in this (@jmacd https://github.com/jmacd @djspoons https://github.com/djspoons @bensigelman https://github.com/bensigelman).
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openzipkin/b3-propagation/issues/4#issuecomment-247508959, or mute the thread https://github.com/notifications/unsubscribe-auth/AAD61_TIRiYMPOe6N5LHlqaymTjU7jVTks5qqgm2gaJpZM4J8TkV .
here's an example of "option 2" which is used by jaeger by @yurishkuro https://github.com/uber/jaeger-client-java/pull/53
@adriancole could the B3 docs state that it's an application concern whether or not the trace ID is returned in the response? i.e. it's not part of the spec and could change at any time so callers should only ever use it as a nice-to-have debugging tool and not rely on it for anything mission critical?
I guess I'm not clear on why returning trace ID limits future flexibility or locks us into anything.
@adriancole https://github.com/adriancole could the B3 docs state that it's an application concern whether or not the trace ID is returned in the response? i.e. it's not part of the spec and could change at any time so callers should only ever use it as a nice-to-have debugging tool and not rely on it for anything mission critical?
For those who are writing B3 explicitly as http middleware, it probably doesn't seem to matter if we copy a header from here to there.
Remember, though, that B3 has never had scope for response, not in sampling or anything else. B3 code in brave and opentracing is extracted to outbound propagation libraries that only affects requests (there are generic extractors here), and are reused across multiple middleware. Whatever we document shouldn't be confusing or imply users should completely remodel their abstractions to now include the backtrace.
IOTW, I'd want to be careful about separating B3 propagation very explicitly from the response, or application/middleware practice or convention. Remember that http is only one of the ways trace data can propagate. In non-http propagation (like thrift, cassandra), these fields are passed binary and have no "X-" prefix.
The part about me personally being uninterested in logging B3-Trace-Id as a response header is mostly not wanting to answer questions about why everyone doesn't do this to higher orders of people. I'm implicitly signed up, right :) When we start documenting things that only one or two do, I end up having to answer why. Then.. guess what happens when we define a real middleware spec, similar to envoy.. I end up having to explain why the response header is now x-request-id? Get me? Selfishness of my time aside, I'm just saying we should think this through and export things very carefully to larger orders of people. None of this limits what individual libraries can do... I mean good specs are built from existing practice. This practice has to come from somewhere, right? :)
anyway....
Want to have a go at drafting what that "current practice" for this might look like? Here's some hints about log correlation (which should be a separate issue if we start taking on generic middleware concerns). Ex in logging, I don't think most use the header name, though what people do is all over the map. Ex some call the log correlation field "traceId" (and don't bother with the span id), some call it "zipkinTraceId".
@adriancole totally understood on wanting to avoid situations that generate timewasting questions, especially on something that doesn't need to be (or shouldn't be) part of the spec. I support whatever you think is best on that front since you have way more experience with community interaction and the consequences of these kinds of documentation decisions.
I was mostly looking to figure out if I needed to change wingtips - the servlet filter helper class currently automatically sets the X-B3-TraceId header on the response. I could change it to return something like X-Wingtips-RequestId if that would be preferable.
@adriancole https://github.com/adriancole totally understood on wanting to avoid situations that generate timewasting questions, especially on something that doesn't need to be (or shouldn't be) part of the spec. I support whatever you think is best on that front since you have way more experience with community interaction and the consequences of these kinds of documentation decisions.
consequences usually are two ways, not just here, more generally I think: perceived complexity (ex people see various half-implemented options and think the project is a mess) or limitations on refactoring (hard to extract request id middleware because of backwards compat).
Worse is being useless due to analysis paralysis. Ex there's a real need to facilitate how to know the trace id, but we do nothing! That's one of the reasons I wanted to enumerate things on this issue.. to see which parts we could support generally, and which could be advisory.
I was mostly looking to figure out if I needed to change wingtips - the servlet filter helper class currently automatically sets the X-B3-TraceId header on the response. I could change it to return something like X-Wingtips-RequestId if that would be preferable.
Personally, I would change it to something like "X-Wingtips-RequestId" or similar because in this case, Wingtips is the edge and that which would add the response header. There could be a future where people do the same thing generically, and then there's a standard header everyone uses (B3, not B3, etc), like "X-Trace.. (wait XTrace is a project).. errr you know what I mean :P Anyway, "X-Wingtips-RequestId" or similar would be cool, especially if it also ties to log correlation practice. In B3, when we discuss adding a request id to the response, we could hint at use cases for it, and examples like Wingtips.
I think the concept of a correlation-id to tie requests together with tracers and logs is not the scope of Zipkin and all about the scope of the consumer / frameworks handling transport details. See https://github.com/openzipkin/openzipkin.github.io/issues/48#issuecomment-251644265
Long story short. It is the concern of the ones building a platform to make sure they originate a correlation identifier at the source of the request and properly handle the propagation of it, so any type of instrumentation backend can record this correlation id with the hooks they provide (binary annotations, contextual logging key-value pairs, etc.). With this we don't introduce scope creep and reliance on something we shouldn't be promising in the first place as it exceeds the domain of tracing.
