Processing errors adds duplicate error.type diagnostic tags to kestrel.connection.duration in Microsoft.AspNetCore.Server.Kestrel

[nsentinel]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the bug

In case of processing errors, Microsoft.AspNetCore.Server.Kestrel diagnostic metrics contain duplicates of error.type in the output statistics for kestrel.connection.duration, both when viewed by standard diagnostic tools and when publishing values to Prometheus/OpenTelemetry, which leads to failures due to duplication error.

The issue is similar to the situation in https://github.com/dotnet/aspnetcore/issues/55159.

Examples of publishing to prometheus:

kestrel_connection_duration_seconds_bucket{otel_scope_name="Microsoft.AspNetCore.Server.Kestrel",error_type="connection_reset",error_type="connection_reset",network_protocol_name="http",network_protocol_version="2",network_transport="tcp",network_type="ipv4",server_address="192.168.1.2",server_port="9559",tls_protocol_version="1.2",le="0.1"} 0 1724805383433

kestrel_connection_duration_seconds_bucket{otel_scope_name="Microsoft.AspNetCore.Server.Kestrel",error_type="keep_alive_timeout",error_type="keep_alive_timeout",network_protocol_name="http",network_protocol_version="2",network_transport="tcp",network_type="ipv4",server_address="192.168.1.2",server_port="9559",tls_protocol_version="1.2",le="0.5"} 0 1724805383433

The application works fine with .NET 8, the issue occurs when migrating to .NET 9 Preview 7

Expected Behavior

The tags on the metrics should not be duplicated

Steps To Reproduce

1. Run the example application below using dotnet run

var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();

app.MapGet("/", (HttpContext context, CancellationToken _) => Console.WriteLine(context.Connection.RemoteIpAddress));
app.Run();

2. Attach using dotnet-counters collect -n DuplicateTagRepro Microsoft.AspNetCore.Server.Kestrel (or any other tooling that allows seeing the Microsoft.AspNetCore.Server.Kestrel metrics.

3. Simulate errors, e.g., by running the following code 1 or more times, adjusting the connection port to match the running application (see step 1)

using System.Net;
using System.Net.Sockets;

using var socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
socket.LingerState = new LingerOption(false, 0);

socket.Connect(IPAddress.Loopback, 62203);
socket.Send("aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"u8);
socket.Close();

4. Observe that metrics now show Microsoft.AspNetCore.Server.Kestrel,kestrel.connection.duration metrics with duplicated error.type=invalid_request_line tags

E.g. Microsoft.AspNetCore.Server.Kestrel,kestrel.connection.duration (s)[error.type=invalid_request_line;error.type=invalid_request_line;network.protocol.name=http;network.protocol.version=1.1;network.transport=tcp;network.type=ipv4;server.address=127.0.0.1;server.port=62203;Percentile=95],Metric,0.00519561767578125

Exceptions (if any)

No response

.NET Version

9.0.100-preview.7.24407.12

Anything else?

No response

[JamesNK]

Thanks for filling the issue. I could reproduce it with your sample and I have a fix here: https://github.com/dotnet/aspnetcore/pull/57561

[nsentinel]

Thanks for the quick fix!