Standard logging integration

[pavolloffay]

There are two possible integrations:

• [ ] put SpanContext into standard logs - there is a problem that OpenTracing does not define getters on SpanContext therefore it's not possible to extract any IDs from it.
• [x] redirect standard logging to OT logs

[pavolloffay]

I am looking into this

[pavolloffay]

https://github.com/opentracing-contrib/java-spring-cloud/pull/97 implemented option 2 - redirect standard logging to Span logs.

Option 1 depends on https://github.com/opentracing/specification/issues/24.

[mdvorak]

I've created implementation that wraps ScopeManager and creates MDC (slf4j) context via user-supplied function. This abstraction can still be added into library, with specific implementation bound to the tracer would have to be supplied. For Jaeger, it can already be created in its autoconfig out-of-the-box.

@pavolloffay If you're interested, I can prepare PR. I would rather have the code here then in separate company-maintained jar.

When opentracing/specification#24 lands, generic handler can be created.

EDIT: This is roughly current implementation https://gist.github.com/mdvorak/80c52b56d2587f66b6201f3cf36c4d1c

[mdvorak]

@pavolloffay Any opinion on this?

[pavolloffay]

It sounds interesting. +1 on my side. @geoand can be the generic stuff added to jaeger starter?

[geoand]

I need to take a look, I'm not familiar with the scope of this. I'll check it out and get back to you soon

[mdvorak]

With current architecture, its a bit complicated. While classes DiagnosticContextScopeManager and SpanDiagnosticContext are generic and could be in core, rest is tracer specific, since there is no way to customize ScopeManager a in generic way. There is not even a way to get current ScopeManager from io.jaegertracing.Tracer.Builder.

In case of Jaeger, JaegerAutoConfiguration could be modified to automatically use ScopeManager bean if available, same as currently Sampler and Reporter. This would allow user to override it without customizer, but still, wrapper class DiagnosticContextScopeManager needs an original ScopeManager, which would have to be provided by tracer library.. Its pretty complex, due to missing APIs, so maybe it would be for the best to simply duplicate same logic for every tracer instead of creating complex shared solution. Would be nice to share those generic classes thou.

[geoand]

This looks like a very nice idea to me. I would think that DiagnosticContextScopeManager and SpanDiagnosticContext would be in one of the generic java opentracing libraries.

Just to clarify, the MDC based implementation only works when the delegate ScopeManager is a ThreadLocalScopeManager (due to MDC also being tied to the current thread), is that correct?

[mdvorak]

You are correct. MDC are also thread based. If you would provide completely custom, not thread bound scopes, I'm not sure it would be possible to handle propagation at all. But class name can reflect that as well.

[geoand]

Thanks @mdvorak for the confirmation.

I really like the solution as long as it's limitations are properly documented

[geoand]

FWIW, Spring Cloud Sleuth also uses the same approach:

https://github.com/spring-cloud/spring-cloud-sleuth/blob/master/spring-cloud-sleuth-core/src/main/java/org/springframework/cloud/sleuth/log/Slf4jCurrentTraceContext.java

[tanumats]

Our team wants to output trace IDs and span IDs propagated by Opentracing clients in the log, but because they are not compatible with the current function, we are very interested in the agenda. What will be the future expansion?

[HoffiMuc]

any news on this? I'd love to use jaeger, but as long as I cannot put trace context (incl. baggage items) inside each and every standard spring boot log line, my hands are tied, unfortunately.

[chengchen]

@HoffiMuc Inspired by the code of @mdvorak, we have implemented this for our own usage. The code is more or less generic, you could just check out the repo and build it yourself. Until now, it works fine for us in production.

[geoand]

@chengchen would you like perhaps to contribute it to this project?

[chengchen]

Hi @geoand The reason why we are not contributing this part of the code is we made some simplifications and assumptions on the usage... If you look at this part of the code, you will notice that we are injecting the return value of SpanContext.toString(). It might look strange to you, but the reason behind is we need to log the parent span id as well, however, the opentracing-api is not exposing it for some reason. And since we are using the jaeger implementation, this toString() method will return a value like {trace-id}:{parent-span-id}:{span-id}:1, which means that using another implementation might not work with this workaround.

[mdvorak]

Note: https://github.com/opentracing/specification/blob/master/rfc/trace_identifiers.md That was original blocker for tracer agnostic solution I believe. It probably won't fit every use-case, as @chengchen mentions (and since every tracer is a bit different, it might not be possible). But it might be "good enough" for most users. I will probably be revisiting this in a month or so, but no promises :)

[geoand]

Thanks folks!

[rubasace]

@chengchen first of all thx for sharing your working solution. Just FYI. I've been facing very similar issues and I've basically based my approach on yours (mainly just care about traceId and spanId, no need for context). In order to override the ScopeManager you can directly create a bean implementing io.opentracing.contrib.java.spring.jaeger.starter.TracerBuilderCustomizer and call .withScopeManager() there, without needing to modify the original autoconfiguration (the original autoconfiguration calls all the customizers. That way you mainly need that Customizer and your ContextScope and ContextScopeManager classes.

Hope it helps if someone is coming looking for a way of logging trace data :)

[roadSurfer]

Unfortunately @chengchen's code is no longer visible/available, so it's not possible for me to see how they solved the problem. I can follow the code here which lets me log the trace & span ids, but unfortunately breaks sending them on to a Jaeger server. Have there been any progress with logging the trace & span ids?

[chengchen]

Hi @roadSurfer unfortunately we have made our repo private because it contains some private credentials and some opinionated choices... If you don't have huge amount of traces or not heavily using Reactor e.g. you could just use this open-sourced solution: https://github.com/jaegertracing/jaeger-client-java/blob/master/jaeger-core/src/main/java/io/jaegertracing/internal/MDCScopeManager.java