feat(tracing): support W3C Trace Context header

[jjangga0214]

Feature Request

What problem are you trying to solve?

Currently, Linkerd only supports B3 Propagation for tracing header. My applications use w3c Trace Context header, conversely.

How should the problem be solved?

Linkerd supports w3c spec.

Any alternatives you've considered?

Currenlty, nope, as I believe w3c spec will be the converged standard.

How would users interact with this feature?

By annotation, user can choose which format to use(b3, or w3c), something like the below.

config.linkerd.io/trace-format: w3c

Thanks!

[adleong]

For reference, this is the relevant proxy code: https://github.com/linkerd/linkerd2-proxy/blob/main/linkerd/trace-context/src/propagation.rs

I don't think an annotation is needed here. If the proxy supported w3c trace context, it should automatically participate in w3c traces when a w3c trace context is present on the request.

[adleong]

@jjangga0214 thanks for filing this. I think that adding w3c trace context support to the proxy would not be too difficult. However, I think the biggest question is around developing a reference architecture that uses w3c trace contexts. (See https://linkerd.io/2/tasks/distributed-tracing/#ingress for some discussion of this)

So I think the first step here would be to develop a reference architecture (including an ingress to initiate traces) which uses w3c trace context. We can then use that reference architecture as a test bed when adding support for w3c trace context to the proxy.

[jjangga0214]

@adleong Thank you so much for the response!

I don't think an annotation is needed here. If the proxy supported w3c trace context, it should automatically participate in w3c traces when a w3c trace context is present on the request.

I agree in genereal.

But, I think, there might be a rare case that some of microservices use b3 propagation and some use w3c trace context header. This might happen, for some reason, when a legacy service creates root span of b3 format, and some propagate it. A company might not be ready to patch the code, for administrative reason, even if it's just only removing the tracing.

config.linkerd.io/trace-collector: my-collector:8080

In this case, linkerd would emits b3 and w3c format both(or only one, or confused?) to the collector. Then the collector responds bad request error, and this would result a full of red lines on log stream.

So, if this case is thought to be too rare that it's not worth considering, then yes, the annotation or anything similar would not be needed.

May I ask Linkerd's thought on this?

Thanks for the reply again, by the way!

[adleong]

One thing that's important to note is that the trace propagation format (such as x-b3, w3c trace context, etc.) is separate from the export format (such as OpenCensus gRPC). If the proxy gets a request with multiple different trace propagation contexts, it would be able to propagate both. It would also be able to export both. This may cause duplicate traces but should not result in errors.

[jwilm]

Hello! I'm very interested in this feature as a large production user of the linkerd2 proxy, and I'm potentially interested in contributing a patch for it. Reading through the notes here, this stood out to me:

So I think the first step here would be to develop a reference architecture (including an ingress to initiate traces) which uses w3c trace context.

Is this a necessary first step? At least for us, it would be incredibly helpful if it was simply possible to propagate w3c trace contexts even if the ingress does not yet initiate the trace context.

[adleong]

Hi @jwilm! My point was more that we'd want some kind of environment where w3c traces are present in order to test against. If you have an environment where the w3c traces are initiated by something other than the ingress, that would work too.

[MeijerM1]

We are also very interested in this feature.

[eugenepaniot]

+1

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[dwilliams782]

Commenting to remove stale label..

[adleong]

In progress: https://github.com/linkerd/linkerd2-proxy/pull/2179

[mateiidavid]

Hi folks! It seems that this has flown under the radar, but W3C header support has been shipped in the proxy (https://github.com/linkerd/linkerd2-proxy/releases/tag/release%2Fv2.190.0). The two latest edge versions should support this.

We will consolidate the tracing docs in the upcoming stable version. Worth knowing, that to propagate w3c contexts, you don't need to do anything configuration wise. The proxy will automatically pick it up if its present on the request, and will sample it according to the header value (i.e make sure your sampling is set correctly :)).

When multiple headers are present: proxy will use w3c by default, if that's not present, it will fallback to b3. It is important to ensure no w3c context header is present if you want to do b3, as the b3 headers will be ignored.

Last but not least, please try it out so we can smoothen any issues before 2.13 is out. Thanks!