[BUG] New SDK blocks HTTP spans

[MatisseHack]

Library name and version

Azure.Core 1.24.0

Describe the bug

I use the OpenTelemetry.Instrumentation.Http package to trace requests made by my app. I was able to see traffic generated by the older Microsoft.Azure. management libraries, but the newer Azure. packages seem to block HTTP spans. I don't see a way to enable HTTP logging with the new libraries.

Expected behavior

HTTP spans should not be suppressed by the Azure SDK or there should be an option to turn off suppression.

Actual behavior

HTTP spans from OpenTelemetry.Instrumentation.Http are not available.

As a workaround, Azure.Core.Http spans are available after enabling Azure.Experimental.EnableActivitySource, but these spans are not as flexible and are not consistent with the other HTTP spans in my app.

Reproduction Steps

1. Use the new Azure. management libraries
2. Enable OpenTelemetry HttpClient instrumentation

Environment

Azure AppService using .NET 6

[azure-sdk]

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'Monitor - Exporter:0.33365932,Azure.Core:0.12701364,Tables:0.06482491'

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[JoshLove-msft]

Hi @MatisseHack, Can you provide an example of a span that you are expecting, but not seeing? Can you clarify why the Azure.Core spans are not sufficient?

@m-nash, @ArthurMa1978 - are we adding individual spans per method call in the new mgmt libraries?

[MatisseHack]

Any spans produced by OpenTelemetry.Instrumentation.Http within the Azure SDK are missing. Here is a simple example:

Activity.Id:          00-22cd49f5eea075f87ef9b72a3816d34b-0f63c7cb033b9e00-01
Activity.ParentId:    00-22cd49f5eea075f87ef9b72a3816d34b-703f7f713b1665ad-01
Activity.ActivitySourceName: OpenTelemetry.Instrumentation.Http
Activity.DisplayName: HTTP PUT
Activity.Kind:        Client
Activity.StartTime:   2022-07-18T19:50:22.1475110Z
Activity.Duration:    00:00:01.1814120
Activity.TagObjects:
    http.method: PUT
    http.host: management.azure.com
    http.url: https://management.azure.com/subscriptions/[subscription-id]/resourcegroups/[rg-name]?api-version=2021-04-01
    http.status_code: 200
    otel.status_code: UNSET
Resource associated with Activity:
    service.name: [service-name]
    service.instance.id: 353a1a26-930f-4d16-a51b-0609f745b4de

We use OpenTelemetry.Instrumentation.Http because we call multiple different APIs and it's easier to query our telemetry data if all HTTP spans are in a consistent format. Using OpenTelemetry.Instrumentation.Http is also more powerful because we can enrich the HTTP spans with extra information when needed. I don't think there is anything similar available for the Azure.Core.Http spans, right?

I do see spans for individual method calls now (e.g. ResourceGroupCollection.CreateOrUpdate), which is a great addition! I think these method spans provide plenty of information for most projects, but ideally it would still be possible to use OpenTelemetry.Instrumentation.Http.

[JoshLove-msft]

Have you followed one of the set up steps outlined here?

Since the OpenTelemetry support is still experimental, you need to set a flag to enable it.

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[MatisseHack]

Yes, I have enabled the experimental OpenTelemetry support. Those spans do work!

The issue is not that I cannot get telemetry from the Azure SDK; the issue is that the Azure SDK suppresses telemetry spans produced by OpenTelemetry.Instrumentation.Http.

The Azure.Core.Http spans are a helpful alternative, but are not sufficient for some use cases and are not consistent with the OpenTelemetry.Instrumentation.Http based HTTP spans I collect from other SDKs in my project.

[JoshLove-msft]

Hi @MatisseHack, Since the Azure.Core library creates its own HTTP spans, the Otel library automatically suppresses creation of these same spans. If you don't want the spans from Azure.Core, you can specify the specific library that you want to collect spans from, e.g.:

AddSource("Azure.ResourceManager.*")

instead of using "Azure.*"

Can you tell us more about what is missing from the Azure.Core HTTP spans?

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[MatisseHack]

Since the Azure.Core library creates its own HTTP spans, the Otel library automatically suppresses creation of these same spans.

Yes exactly. I would like an option to turn this behavior off so that I can collect standard HTTP spans instead of the Azure SDK's custom implementation.

If you don't want the spans from Azure.Core, you can specify the specific library that you want to collect spans from

I do want HTTP spans. I would prefer to use spans from OpenTelemetry.Instrumentation.Http, but they aren't available so spans from Azure.Core.Http are my only option.

Can you tell us more about what is missing from the Azure.Core HTTP spans?

There are two reasons I prefer OpenTelemetry.Instrumentation.Http over Azure.Core.Http:

1. It's an inconvenience to have multiple types of HTTP spans in my app. If every SDK in my app produced their own custom HTTP spans, I'd have an unintelligible mess of HTTP spans that would be difficult to query. The spans from OpenTelemetry.Instrumentation.Http provide a uniform implementation that works everywhere except the Azure SDK.

2. The main blocker for my team is that the Azure SDK does not support enriching HTTP spans. OpenTelemetry.Instrumentation.Http provides a convenient mechanism to enrich HTTP spans with extra information that is not collected by default. I don't see an equivalent capability for Azure.Core.Http.

[MatisseHack]

To illustrate my desired behavior, take a look at the AWS SDK, which is a good example of how HTTP spans could work in the Azure SDK. By default, the AWS SDK doesn't produce any spans of its own and OpenTelemetry.Instrumentation.Http spans work normally. After adding an instrumentation library, the AWS SDK produces the equivalent of Azure's Azure.ResourceManager.* spans and suppresses OpenTelemetry.Instrumentation.Http spans. I'm not sure suppressing child spans by default is the best choice, but at least there is a configuration option to disable suppression of child spans entirely.

I would like the Azure SDK to work in a similar way. If Azure.Experimental.EnableActivitySource is disabled, OpenTelemetry.Instrumentation.Http spans should not be suppressed. When Azure.Experimental.EnableActivitySource is enabled, downstream spans should still be accessible (potentially behind a configuration option).

[JoshLove-msft]

@MatisseHack, the Azure library is not suppressing the OpenTelemetry spans, but rather the OpenTelemetry library is not creating duplicate HTTP spans. My question was more around why do you prefer the OpenTelemetry HTTP spans to the Azure.Core Http spans?

Edit: I missed your previous comment. Thanks for clarifying what is missing in the Azure spans vs OpenTelemetry. Also, I see your point about not wanting to use different HTTP spans across your app if you have some Azure libraries and other AWS libraries, as an example. As for adding hooks to enrich Azure.Core HTTP spans, I will follow up on this.

In terms of having HTTP spans off by default, they sort of already are, because you have to do the following to turn them on:

using var openTelemetry = Sdk.CreateTracerProviderBuilder()
    .AddSource("Azure.*") // Collect all traces from Azure SDKs

If you want to exclude the HTTP traces, but still want the library method traces, you could do the following:

using var openTelemetry = Sdk.CreateTracerProviderBuilder()
   .AddSource("Azure.ResourceManager.*") // Collect all traces from Azure SDKs

[MatisseHack]

It definitely makes sense that I need to opt in to collecting Azure.* spans! I'd just like to minimize interference with with other telemetry sources I've subscribed to. e.g. builder.AddSource("OpenTelemetry.Instrumentation.Http"). However, I'm not too concerned about default behavior as long as it's configurable.

[JoshLove-msft]

As for adding hooks to enrich Azure.Core HTTP spans, I will follow up on this.

@lmolkova do you know if this is possible?

[JoshLove-msft]

@MatisseHack, if you don't opt in to collecting the "Azure.*" spans, don't you get the Http spans as you are expecting?

[github-actions[bot]]

Hi @MatisseHack. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[lmolkova]

@MatisseHack thanks for reporting this!

As @JoshLove-msft pointed out you don't have to opt into Azure SDK HTTP spans - it is possible to subscribe to specific sources, e.g. AddSource("Azure.ResourceManager.*"). The name matches root namespace of the package.

With the new OTel instrumentation on .NET 7, underlying HTTP spans are no longer suppressed (this is not in the control of Azure SDK). By not enabling Azure.Core.Http source you should only get default instrumentations.

[github-actions[bot]]

Hi @MatisseHack, we're sending this friendly reminder because we haven't heard back from you in 7 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[MatisseHack]

Yes, after updating .NET and the OpenTelemetry and Azure SDKs I'm now getting the OpenTelemetry.Instrumentation.Http spans again! Not sure exactly which update did the trick, but it's working as expected now. Thanks for the discussion and help!