Test failure System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt

[VincentBu]

trafficstars

Test affected:

• CancelPendingRequest_DropsStalledConnectionAttempt - System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt

Failures 4/1-8/3 (incl. PRs):

• 57 failures (~8 per day), started on 7/27 only in official runs - amd64, arm and arm64

let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests('', 'CancelPendingRequest_DropsStalledConnectionAttempt', true, 'RemoteExecutionException', true);

Original report

Run: runtime-libraries-coreclr outerloop 20220727.7

Failed test：

net7.0-Linux-Release-arm-CoreCLR_release-(Debian.11.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:debian-11-helix-arm32v7-20210304164347-5a7c380

- System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt(versionString: \"2.0\", firstConnectionDelayMs: 10000, requestTimeoutMs: 1000, pendingConnectionTimeoutOnRequestCompletion: 100)

net7.0-Linux-Release-arm-CoreCLR_release-(Debian.10.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:debian-10-helix-arm32v7-20210304164340-6616c63

- System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt(versionString: \"2.0\", firstConnectionDelayMs: 10000, requestTimeoutMs: 1000, pendingConnectionTimeoutOnRequestCompletion: 100)

net7.0-Linux-Release-x64-CoreCLR_release-Ubuntu.2204.Amd64.Open

- System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt(versionString: \"2.0\", firstConnectionDelayMs: 10000, requestTimeoutMs: 1000, pendingConnectionTimeoutOnRequestCompletion: 100)

Error message:

Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Remote process failed with an unhandled exception.


Stack trace

Child exception:
  System.AggregateException: One or more errors occurred. (A task was canceled.) ( Received an unexpected EOF or 0 bytes from the transport stream.)
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 632
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 551
   at System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.<>c__DisplayClass2_0.<<CancelPendingRequest_DropsStalledConnectionAttempt_Impl>b__0>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Cancellation.NonParallel.cs:line 66
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 88
   at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext() in /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs:line 38
--- End of stack trace from previous location ---
   at System.Net.Test.Common.Http2LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 234
   at System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt_Impl(String versionString, String firstConnectionDelayMsString, String requestTimeoutMsString) in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Cancellation.NonParallel.cs:line 52
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Program.cs:line 65
 ---> (Inner Exception #1) System.IO.IOException:  Received an unexpected EOF or 0 bytes from the transport stream.
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 363
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 289
   at System.Net.Test.Common.Http2LoopbackConnection.CreateAsync(SocketWrapper socket, Stream stream, Http2Options httpOptions, TimeSpan timeout) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 72
   at System.Net.Test.Common.Http2LoopbackServer.AcceptConnectionAsync(Nullable`1 timeout) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 99
   at System.Net.Test.Common.Http2LoopbackServer.EstablishConnectionGetSettingsAsync(Nullable`1 timeout, Nullable`1 ackTimeout, SettingsEntry[] settingsEntries) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 130
   at System.Net.Test.Common.Http2LoopbackServer.EstablishConnectionAsync(Nullable`1 timeout, Nullable`1 ackTimeout, SettingsEntry[] settingsEntries) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 119
   at System.Net.Test.Common.Http2LoopbackServer.AcceptConnectionAsync(Func`2 funcAsync) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 159
   at System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.<>c.<<CancelPendingRequest_DropsStalledConnectionAttempt_Impl>b__2_1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Cancellation.NonParallel.cs:line 71
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120<---


Child process:
  System.Net.Http.Functional.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel System.Threading.Tasks.Task CancelPendingRequest_DropsStalledConnectionAttempt_Impl(System.String, System.String, System.String)

Child arguments:
  2.0, 10000, 1000

[msftbot[bot]]

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run: runtime-libraries-coreclr outerloop 20220727.7

Failed test：

net7.0-Linux-Release-arm-CoreCLR_release-(Debian.11.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:debian-11-helix-arm32v7-20210304164347-5a7c380

- System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt(versionString: \"2.0\", firstConnectionDelayMs: 10000, requestTimeoutMs: 1000, pendingConnectionTimeoutOnRequestCompletion: 100)

net7.0-Linux-Release-arm-CoreCLR_release-(Debian.10.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:debian-10-helix-arm32v7-20210304164340-6616c63

- System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt(versionString: \"2.0\", firstConnectionDelayMs: 10000, requestTimeoutMs: 1000, pendingConnectionTimeoutOnRequestCompletion: 100)

net7.0-Linux-Release-x64-CoreCLR_release-Ubuntu.2204.Amd64.Open

- System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt(versionString: \"2.0\", firstConnectionDelayMs: 10000, requestTimeoutMs: 1000, pendingConnectionTimeoutOnRequestCompletion: 100)

Error message:

Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Remote process failed with an unhandled exception.


Stack trace

Child exception:
  System.AggregateException: One or more errors occurred. (A task was canceled.) ( Received an unexpected EOF or 0 bytes from the transport stream.)
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 632
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) in /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs:line 551
   at System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.<>c__DisplayClass2_0.<<CancelPendingRequest_DropsStalledConnectionAttempt_Impl>b__0>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Cancellation.NonParallel.cs:line 66
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 88
   at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext() in /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs:line 38
--- End of stack trace from previous location ---
   at System.Net.Test.Common.Http2LoopbackServerFactory.CreateServerAsync(Func`3 funcAsync, Int32 millisecondsTimeout, GenericLoopbackOptions options) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 234
   at System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt_Impl(String versionString, String firstConnectionDelayMsString, String requestTimeoutMsString) in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Cancellation.NonParallel.cs:line 52
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args) in /_/src/Microsoft.DotNet.RemoteExecutor/src/Program.cs:line 65
 ---> (Inner Exception #1) System.IO.IOException:  Received an unexpected EOF or 0 bytes from the transport stream.
   at System.Net.Security.SslStream.ReceiveBlobAsync[TIOAdapter](CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 363
   at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](Boolean receiveFirst, Byte[] reAuthenticationData, CancellationToken cancellationToken) in /_/src/libraries/System.Net.Security/src/System/Net/Security/SslStream.IO.cs:line 289
   at System.Net.Test.Common.Http2LoopbackConnection.CreateAsync(SocketWrapper socket, Stream stream, Http2Options httpOptions, TimeSpan timeout) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackConnection.cs:line 72
   at System.Net.Test.Common.Http2LoopbackServer.AcceptConnectionAsync(Nullable`1 timeout) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 99
   at System.Net.Test.Common.Http2LoopbackServer.EstablishConnectionGetSettingsAsync(Nullable`1 timeout, Nullable`1 ackTimeout, SettingsEntry[] settingsEntries) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 130
   at System.Net.Test.Common.Http2LoopbackServer.EstablishConnectionAsync(Nullable`1 timeout, Nullable`1 ackTimeout, SettingsEntry[] settingsEntries) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 119
   at System.Net.Test.Common.Http2LoopbackServer.AcceptConnectionAsync(Func`2 funcAsync) in /_/src/libraries/Common/tests/System/Net/Http/Http2LoopbackServer.cs:line 159
   at System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.<>c.<<CancelPendingRequest_DropsStalledConnectionAttempt_Impl>b__2_1>d.MoveNext() in /_/src/libraries/System.Net.Http/tests/FunctionalTests/SocketsHttpHandlerTest.Cancellation.NonParallel.cs:line 71
--- End of stack trace from previous location ---
   at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) in /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs:line 120<---


Child process:
  System.Net.Http.Functional.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel System.Threading.Tasks.Task CancelPendingRequest_DropsStalledConnectionAttempt_Impl(System.String, System.String, System.String)

Child arguments:
  2.0, 10000, 1000

Author:	VincentBu
Assignees:	-
Labels:	area-System.Net.Http
Milestone:	-

[karelz]

Triage: Looks like a recent regression -- happening 8x per day. We should disable the test ASAP - @rzikm can you please do it? Thanks!

[antonfirsov]

This is test from #71785. It's only happening on Ubuntu and only for HTTP 2.0.

[wfurt]

net7.0-Linux-Release-arm-CoreCLR_release-(Debian.11.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:debian-11-helix-arm32v7-20210304164347-5a7c380

- System.Net.Http.Functional.Tests.SocketsHttpHandler_Cancellation_Test_NonParallel.CancelPendingRequest_DropsStalledConnectionAttempt(versionString: \"2.0\", firstConnectionDelayMs: 10000, requestTimeoutMs: 1000, pendingConnectionTimeoutOnRequestCompletion: 100)

while Ubuntu is base image, the test is actually running on Debian.

[shimingsg]

Failed again in: runtime-coreclr libraries-jitstress 20220804.1

Failed test:

net7.0-Linux-Release-arm-CoreCLR_checked-no_tiered_compilation-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440
- System.Net.Security.Tests.NegotiateAuthenticationKerberosTest.Loopback_Success

net7.0-Linux-Release-arm-CoreCLR_checked-jitminopts-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440
- System.Net.Security.Tests.NegotiateAuthenticationKerberosTest.Loopback_Success

net7.0-Linux-Release-arm-CoreCLR_checked-tailcallstress-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440
- System.Net.Security.Tests.NegotiateAuthenticationKerberosTest.Loopback_Success

net7.0-Linux-Release-arm-CoreCLR_checked-zapdisable-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440
- System.Net.Security.Tests.NegotiateAuthenticationKerberosTest.Loopback_Success

net7.0-Linux-Release-arm-CoreCLR_checked-jitstress1-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440
- System.Net.Security.Tests.NegotiateAuthenticationKerberosTest.Loopback_Success

net7.0-Linux-Release-arm-CoreCLR_checked-jitstress1-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440
- System.Net.Security.Tests.NegotiateAuthenticationKerberosTest.Loopback_Success

net7.0-Linux-Release-arm-CoreCLR_checked-jitstress1_tiered-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440
- System.Net.Security.Tests.NegotiateAuthenticationKerberosTest.Loopback_Success

net7.0-Linux-Release-arm-CoreCLR_checked-jitstress2-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440
- System.Net.Security.Tests.NegotiateAuthenticationKerberosTest.Loopback_Success

net7.0-Linux-Release-arm-CoreCLR_checked-jitstress2_tiered-(Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440
- System.Net.Security.Tests.NegotiateAuthenticationKerberosTest.Loopback_Success

Error message:

Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Remote process failed with an unhandled exception.


Stack trace

Child exception:
  Xunit.Sdk.TrueException: Server authentication failed with GenericFailure
Expected: True
Actual:   False
   at System.Net.Security.Tests.NegotiateAuthenticationKerberosTest.<>c.<Loopback_Success>b__2_0() in /_/src/libraries/System.Net.Security/tests/FunctionalTests/NegotiateAuthenticationKerberosTest.cs:line 52
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodInvoker.cs:line 64

Child process:
  System.Net.Security.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 System.Net.Security.Tests.NegotiateAuthenticationKerberosTest+<>c Void <Loopback_Success>b__2_0()```

[antonfirsov]

@shimingsg that is a different, unrelated test: #73343

[karelz]

Test disabled in PR #73300

[karelz]

Test is disabled, not blocking CI anymore - removing 'blocking-outerloop' label.