With AspNetCore 6, OtlpExporter fail to send trace if running on linux

[mrioux3]

Bug Report

OpenTelemetry 1.3.0 OpenTelemetry.Exporter.OpenTelemetryProtocol 1.3.0 OpenTelemetry.Extensions.Hosting 1.0.0-rc9.4 OpenTelemetry.Instrumentation.AspNetCore 1.0.0-rc9.4

Runtime version net6.0

Symptom

When running an AspNetCore 6 on linux, the OtlpExporter fail to send the traces to the exporter with no error message.

What is the expected behavior?

The trace should be sent to the otlpExporter when running under linux.

What is the actual behavior?

The trace is never sent when running with linux.

Reproduce

This is based on the code example from https://opentelemetry.io/docs/instrumentation/net/getting-started/

builder.Services.AddOpenTelemetryTracing(tracerProviderBuilder =>
{
    tracerProviderBuilder
                .AddSource(ServiceName)
                .SetResourceBuilder(ResourceBuilder.CreateDefault()
                        .AddService(serviceName: ServiceName, serviceVersion: ServiceVersion))
                .AddAspNetCoreInstrumentation()
                .AddOtlpExporter(opt =>
                {
                    opt.Endpoint = new Uri(endpoint);
                    opt.Protocol = OtlpExportProtocol.Grpc;
                });
});

var app = builder.Build();

var MyActivitySource = new ActivitySource(ServiceName, ServiceVersion);

app.MapGet("/hello", () =>
{
    // Track work inside of the request
    using var activity = MyActivitySource.StartActivity("Test");
    activity?.SetTag("Test", 100);

    return activity?.RootId;
});

Additional Context

This issue only happen on linux. The code work on Windows. This was tested on "Ubuntu 20.04.4 LTS" version "5.4.0-1055-kvm" and "5.10.16.3-microsoft-standard-WSL2"

Using the consoleexporter, I see the data showing up in the console and everything seems fine.

I also try to change the opentelemetry nuget to 1.2.0 and 1.4.0-alpha.2 and still had the issues with those versions. (the other nugets were updated to match)

[mrblonde91]

For the exporter, can you try specifying the endpoint like so.

 otlpOptions.Endpoint = new Uri(endpoint + "/api/v1/traces"); 

[mrioux3]

For the exporter, can you try specifying the endpoint like so.

 otlpOptions.Endpoint = new Uri(endpoint + "/api/v1/traces"); 

I tried and that didn't work.

[cijothomas]

https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/src/OpenTelemetry.Exporter.OpenTelemetryProtocol#troubleshooting Follow the troubleshooting steps and can you get the logs, so we can figure out whats going on?

[mrioux3]

Trying the self diagnostic in my main application, the only event that show up is System.Diagnostics.Tracing.EventPayload : 2022-09-01T19:58:56.8518308Z:{System.Diagnostics.Tracing.EventPayload} 2022-09-01T19:58:56.8518360Z:{System.Diagnostics.Tracing.EventPayload} 2022-09-01T19:58:56.8518408Z:{System.Diagnostics.Tracing.EventPayload}

The log file is filled with this event.

[VMelnalksnis]

@cijothomas I'm having similiar issues, where publishing the application with -r win10-x64 everything works as expected, but with -r linux-x64 nothing is exported. I also tried creating the OTEL_DIAGNOSTICS.json file, but that also worked only on windows. Any ideas what could be the issue?

.NET SDK version: 7.0.202 Packages:

OpenTelemetry.Exporter.OpenTelemetryProtocol.Logs 1.4.0-rc.4
OpenTelemetry.Exporter.OpenTelemetryProtocol 1.4.0
OpenTelemetry.Extensions.Hosting 1.4.0
OpenTelemetry.Instrumentation.AspNetCore 1.0.0-rc9.14
OpenTelemetry.Instrumentation.Http 1.0.0-rc9.14
OpenTelemetry.Instrumentation.Process 0.5.0-beta.2
OpenTelemetry.Instrumentation.SqlClient 1.0.0-rc9.13
OpenTelemetry.Instrumentation.Runtime 1.1.0-rc.2

[cijothomas]

@utpilla Please take a look!

[cijothomas]

Trying the self diagnostic in my main application, the only event that show up is System.Diagnostics.Tracing.EventPayload : 2022-09-01T19:58:56.8518308Z:{System.Diagnostics.Tracing.EventPayload} 2022-09-01T19:58:56.8518360Z:{System.Diagnostics.Tracing.EventPayload} 2022-09-01T19:58:56.8518408Z:{System.Diagnostics.Tracing.EventPayload}

The log file is filled with this event.

https://github.com/open-telemetry/opentelemetry-dotnet/pull/5046 fixes the issue!

[cijothomas]

https://github.com/open-telemetry/opentelemetry-dotnet/pull/5046 fixes the self-diagnostics, which will help understand the root cause of the issue reported here.

[aburok]

We have same problem in our application. Even after updating from .net 6.0 to .net 8.0. Application is using AspNetCore.

Packages:

OpenTelemetry(1.70)
OpenTelemetryApi(1.70) 
OpenTelemetryApi.ProviderBuilderExtensions(1.70) 
OpenTelemetry.Exporter.OpenTelemetryProtocol(1.70) 
OpenTelemetry.Exporter.PrometheusAspNetCore(1.7.0-rc.1)
OpenTelemetry.Extensions.Hosting(1.70) 
OpenTelemetry.lnstrumentation.AspNetCore(1.70) 
OpenTelemetry.lnstrumentation.Http(1.70) 
OpenTelemetry.lnstrumentation.Process(0.5.0-beta.3) 
OpenTelemetry.lnstrumentation.Runtime(1.70) 

Configuration:

.AddOpenTelemetry()
.WithTracing(
	tracerProviderBuilder =>
		tracerProviderBuilder
			.AddSource(opts.ServiceName)
			.SetSampler(new AlwaysOnSampler())
			.AddAspNetCoreInstrumentation((options) =>
			{
				options.RecordException = true;
			})
			.AddHttpClientInstrumentation((options =>
			{
				options.RecordException = true;
			}))
			.AddOtlpExporter())

After adding OTEL_DIAGNOSTICS.json and setting log level to Verbose I can see that Activities start and end:

2024-01-11T12:28:34.1408035Z:Activity started. Name = '{0}', Id = '{1}'.{System.Net.Http.HttpRequestOut}{00-37c74d2fc18b6f9716f083254a21d5b3-a5a7b995310df765-01}
2024-01-11T12:28:34.1476990Z:Activity stopped. Name = '{0}', Id = '{1}'.{POST}{00-37c74d2fc18b6f9716f083254a21d5b3-a5a7b995310df765-01}
2024-01-11T12:28:34.1983849Z:Activity started. Name = '{0}', Id = '{1}'.{System.Net.Http.HttpRequestOut}{00-37c74d2fc18b6f9716f083254a21d5b3-717228a8dff2f2f9-01}
2024-01-11T12:28:34.2039340Z:Activity stopped. Name = '{0}', Id = '{1}'.{POST}{00-37c74d2fc18b6f9716f083254a21d5b3-717228a8dff2f2f9-01}
2024-01-11T12:28:34.2329163Z:Activity started. Name = '{0}', Id = '{1}'.{#####}{00-37c74d2fc18b6f9716f083254a21d5b3-0382e85fd4bcba28-01}
2024-01-11T12:28:34.2446729Z:Activity started. Name = '{0}', Id = '{1}'.{System.Net.Http.HttpRequestOut}{00-37c74d2fc18b6f9716f083254a21d5b3-22fc042a507dee41-01}
2024-01-11T12:28:35.2179964Z:Activity stopped. Name = '{0}', Id = '{1}'.{POST}{00-37c74d2fc18b6f9716f083254a21d5b3-22fc042a507dee41-01}
2024-01-11T12:28:35.2223925Z:Activity stopped. Name = '{0}', Id = '{1}'.{#####}{00-37c74d2fc18b6f9716f083254a21d5b3-0382e85fd4bcba28-01}
2024-01-11T12:28:35.2726035Z:Activity stopped. Name = '{0}', Id = '{1}'.{POST calling/v1/callouts}{00-37c74d2fc18b6f9716f083254a21d5b3-c38e450d63a0e3b6-01}

From other app on same host, written in .net 8.0 with Minimal API, with same configuration, tracing IS working.

[aburok]

Workaround

Solution that worked for me was to switch to Zipkin Exporter, which exports spans/activities as JSON over HTTP.

tracerProviderBuilder.AddZipkinExporter(builder =>
{
    builder.Endpoint = new Uri("http://localhost:9411/api/v2/spans");
});

We are using open telemetry collectors on Kubernetes which allows to configure a zipkin receiver.

Open telemetry collector transform the Zipkin format into in internal open telemetry format and can push that span/acitivity to any other desitnation

[cijothomas]

@aburok Can you collect the self diagnostics logs for at least 1 minute and share?

[aburok]

@cijothomas unfortunately I was not able to gathet any open telemetry self diagnostic logs on linux. Even when add OTEL_DIAGNOSTICS.json file in application directory and setting log level to debug, log files do not appear in the linux container. I have made sure that application has right to write to file system.

Self diagnostic logs work fine on windows though. But on windows the problem with exporter does not occur.

[cijothomas]

Weird that the logs are not captured with self-diag!

Could you use an alternate tool to collect the logs? https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-trace is cross platform and can be used to collect logs.

The following should do the trick!

dotnet-trace collect --process-id <PID> --providers OpenTelemetry-Exporter-OpenTelemetryProtocol,OpenTelemetry-Sdk

[aburok]

@cijothomas I have run a dotnet-trace command and have following trace info for both Zipkin and Otlp Exporters:

Is this info enough for you ? Also it would be useful to have logs for OpenTelemetry working on Linux docker containers. So far I was not able to make them work on Linux containers. Logs for OpenTelemetry worked fine on Windows though.

[cijothomas]

Is this info enough for you ? No. Need to get the actual logs..Can you attach the logs file? The screenshot is showing perf graph only, which won't help with understanding why OTLPExporter is failing!

[aburok]

I have found log:

Provider Name/Event Name	Text	Timestamp (ms)
OpenTelemetry-Exporter-OpenTelemetryProtocol/FailedToReachCollector	[rawCollectorUri, http://opentelemetry_collector:4317/v1/traces], [ex, System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
 ---> System.Net.Sockets.SocketException (104): Connection reset by peer
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.Http.HttpConnection.FillAsync(Boolean async)
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean async, Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.SocketsHttpHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClientHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpMessageInvoker.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.Send(HttpRequestMessage request, CancellationToken cancellationToken)
   at OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.BaseOtlpHttpExportClient`1.SendExportRequest(TRequest request, CancellationToken cancellationToken)]	12725

Btw I am using http/protobuf as a protocol.

The collector is up and running with following configuration:

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]

No traces are reaching the collector.

[aburok]

I have finally managed to make it working: setting protocol to http/protobuf and setting the endpoint to OTEL_EXPORTER_OTLP_ENDPOINT=http://opentelemetry_collector:4318/v1/traces did the trick.

[cijothomas]

Closing this issue as the issue is resolved. Please reopen/create a new one, if further help is needed.