Malformed tracestate header breaks Activity.Baggage

[JIntrocaso]

Bug Report

List of all OpenTelemetry NuGet packages and version that you are using (e.g. OpenTelemetry 1.0.2):

OpenTelemetry 1.5.1 OpenTelemetry" 1.5.1" OpenTelemetry.Exporter.Console 1.5.1 OpenTelemetry.Exporter.OpenTelemetryProtocol 1.5.1 OpenTelemetry.Extensions.Hosting 1.5.1 OpenTelemetry.Instrumentation.AspNetCore 1.0.0-rc9.14 OpenTelemetry.Instrumentation.Http 1.0.0-rc9.14 OpenTelemetry.Instrumentation.Runtime 1.5.0

• TBD

Runtime version (e.g. net462, net48, netcoreapp3.1, net6.0 etc. You can find this information from the *.csproj file): dotnet 7

• TBD

Symptom

A clear and concise description of what the bug is. It seems that when an http request goes from one service to another with a tracestate that is missing a tenent id and beginning with the @ character, the dotnet Activity.Baggage is not propagated from one service to the other.

What is the expected behavior?

What did you expect to see? The dotnet Open Telemetry library and instrumentation should not break the functionality of Activity.Baggage values propagating across service calls.

What is the actual behavior?

What did you see instead? When passed a tracestate that is missing a tenent id, Activity.Baggage has no values

Reproduce

Create a self-contained project using the template of your choice, apply the minimum required code to result in the issue you're observing. Repo

There is a console app that will call the gateway service with the following variations:

• Passing a malformed tracestate that will call the service instrumented with open telemetry packages
• Passing a proper tracestate that will call the service instrumented with open telemetry packages
• Passing a malformed tracestate that will call the service not instrumented with open telemetry
• Passing a proper tracestate that will call the service not instrumented with open telemetry

These calls will log the headers and any baggage values in the console of the called services

When examining the logs you will be able to see where the baggage will be properly passed in all calls except for the malformed tracestate being passed to the service that is using the open telemetry packages.

We will close this issue if:

• The repro project you share with us is complex. We can't investigate custom projects, so don't point us to such, please.
• If we can not reproduce the behavior you're reporting.

Additional Context

Add any other context about the problem here.

[cijothomas]

Activity.Baggage is not OpenTelemetry Baggage. See docs for OpenTelemetry baggage: https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/src/OpenTelemetry.Api#baggage-api

Issues with Activity.Baggage can be reported to dotnet/runtime repo directly.

If the issue is present in OpenTelemetry Baggage, please re-open the issue, please share a minimal repro.

[JIntrocaso]

The issue doesn't seem to be with Activity.Baggage except for when I am using the OpenTelemetry library. I'd be happy to pare down the provided repro if you could point out what might be superfluous in the one I provided. I feel like the library shouldn't break propagation if there is a malformed tracestate header.

When looking at the documentation about Baggage provided, I see where it is recommended not to use Activity.Baggage, but that is also making the assumption that all services are using OpenTelemetry which is not our case. We are incrementally adopting the libraries over our services.

[cijothomas]

The issue doesn't seem to be with Activity.Baggage except for when I am using the OpenTelemetry library.

It seems that when an http request goes from one service to another with a tracestate that is missing a tenent id and beginning with the @ character, the dotnet Activity.Baggage is not propagated from one service to the other.

OpenTelemetry components shipped from this repo does not do anything with Activity.Baggage. And as from as I can tell, Activity.Baggage is never influenced by TraceState either.

I do not have the bandwidth to fully go through the shared repro. You can help by narrowing down the issue scope with a smaller repro.

[JIntrocaso]

It seems that when an http request goes from one service to another with a tracestate that is missing a tenent id and beginning with the @ character, the dotnet Activity.Baggage is not propagated from one service to the other.

Right, but the repro demonstrates that it is only broken when using the open telemetry dotnet library. That is where I feel that the issue lies within this repo. The repro is as small as it can be in order to demonstrate the side by side with and without otel library behaviors. Is it too large from a file size, number of projects? Some clarification would be fantastic. I am really trying to be a good open source steward here, and would just really appreciate some clarification for why it is too large.

[cijothomas]

Is it too large from a file size, number of projects? Some clarification would be fantastic.

There are a lot of files when I took a quick glance

Like I said in my previous reply - I do not have the bandwidth to fully go through the shared repro. You can help by narrowing down the issue scope with a smaller repro.

I'll re-open the issue, to see if anyone else has the bandwidth to look at it.

[github-actions[bot]]

This issue was marked stale due to lack of activity and will be closed in 7 days. Commenting will instruct the bot to automatically remove the label. This bot runs once per day.

[github-actions[bot]]

Closed as inactive. Feel free to reopen if this issue is still a concern.