Sockets tests SendFileGetsCanceledByDispose and AcceptGetsCanceledByDispose - throw NotSupportedException - on AppleTV/iOS

[karelz]

Regression as of 8/4 - happens in each Rolling run and some PRs with 7, 14, or 21 failures (based on which legs ran):

Affected tests:

• SendFileGetsCanceledByDispose
  • System.Net.Sockets.Tests.SendFile_Apm
  • System.Net.Sockets.Tests.SendFile_CancellableTask
  • System.Net.Sockets.Tests.SendFile_Task
• AcceptGetsCanceledByDispose
  • System.Net.Sockets.Tests.AcceptApm
  • System.Net.Sockets.Tests.AcceptCancellableTask
  • System.Net.Sockets.Tests.AcceptEap
  • System.Net.Sockets.Tests.AcceptTask

Failure:

   at System.Net.Sockets.Socket.SendPacketsAsync(SocketAsyncEventArgs e, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendPacketsAsync(Socket socket, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.SendFileAsync(String fileName, ReadOnlyMemory`1 preBuffer, ReadOnlyMemory`1 postBuffer, TransmitFileOptions flags, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.SendFileAsync(String fileName, CancellationToken cancellationToken)
   at System.Net.Sockets.Tests.SocketHelperCancellableTask.SendFileAsync(Socket s, String fileName)
   at System.Net.Sockets.Tests.SocketTestHelperBase`1[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].SendFileAsync(Socket s, String fileName)
   at System.Net.Sockets.Tests.SendFile`1.<>c__DisplayClass10_1.<<SendFileGetsCanceledByDispose>b__2>d[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext()
--- End of stack trace from previous location ---
   at System.Net.Sockets.Tests.SendFile`1.<>c__DisplayClass10_0.<<SendFileGetsCanceledByDispose>b__0>d[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext()
--- End of stack trace from previous location ---
   at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
   at System.Net.Sockets.Tests.SendFile`1.<SendFileGetsCanceledByDispose>d__10[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext()
--- End of stack trace from previous location ---

let failedTests = (testNameSubstring : string, methodName : string, messageSubstr: string, includePR : bool, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | where Branch <> 'refs/pull/73374/merge'
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests(
    'System.Net.Sockets.Tests', //testNameSubstring
    '', //methodName
    'NotSupportedException', //messageSubstr
    true,  //includePR
    true); //includePassedOnRerun

[msftbot[bot]]

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details

---

Regression as of 8/4 - happens in each Rolling run and some PRs with 7, 14, or 21 failures (based on which legs ran):

Affected tests:

• SendFileGetsCanceledByDispose
  • System.Net.Sockets.Tests.SendFile_Apm
  • System.Net.Sockets.Tests.SendFile_CancellableTask
  • System.Net.Sockets.Tests.SendFile_Task
• AcceptGetsCanceledByDispose
  • System.Net.Sockets.Tests.AcceptApm
  • System.Net.Sockets.Tests.AcceptCancellableTask
  • System.Net.Sockets.Tests.AcceptEap
  • System.Net.Sockets.Tests.AcceptTask

Failure:

   at System.Net.Sockets.Socket.SendPacketsAsync(SocketAsyncEventArgs e, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendPacketsAsync(Socket socket, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.SendFileAsync(String fileName, ReadOnlyMemory`1 preBuffer, ReadOnlyMemory`1 postBuffer, TransmitFileOptions flags, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.SendFileAsync(String fileName, CancellationToken cancellationToken)
   at System.Net.Sockets.Tests.SocketHelperCancellableTask.SendFileAsync(Socket s, String fileName)
   at System.Net.Sockets.Tests.SocketTestHelperBase`1[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].SendFileAsync(Socket s, String fileName)
   at System.Net.Sockets.Tests.SendFile`1.<>c__DisplayClass10_1.<<SendFileGetsCanceledByDispose>b__2>d[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext()
--- End of stack trace from previous location ---
   at System.Net.Sockets.Tests.SendFile`1.<>c__DisplayClass10_0.<<SendFileGetsCanceledByDispose>b__0>d[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext()
--- End of stack trace from previous location ---
   at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
   at System.Net.Sockets.Tests.SendFile`1.<SendFileGetsCanceledByDispose>d__10[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext()
--- End of stack trace from previous location ---

let failedTests = (testNameSubstring : string, methodName : string, messageSubstr: string, includePR : bool, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | where Branch <> 'refs/pull/73374/merge'
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests(
    'System.Net.Sockets.Tests', //testNameSubstring
    '', //methodName
    'NotSupportedException', //messageSubstr
    true,  //includePR
    true); //includePassedOnRerun

Author:	karelz
Assignees:	-
Labels:	area-System.Net.Sockets, blocking-clean-ci, os-ios, os-tvos
Milestone:	-

[karelz]

These tests are significantly impacting Rolling runs and CI overall. Please disable the tests ASAP.

Note that similar errors first appeared in unmerged PRs: #73258 and #73374 which both seem related -- @directhex perhaps you might know what's going on?

cc @steveisok

[MihaZupan]

Potentially related: #72000 cc: @tmds

[tmds]

   at System.Net.Sockets.Socket.SendPacketsAsync(SocketAsyncEventArgs e, CancellationToken cancellationToken)

I guess this is at:

https://github.com/dotnet/runtime/blob/99297533df71e0d9e38ccf4a6bb6afd236a5f665/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs#L2981-L2984

So on these platforms the new cases added in https://github.com/dotnet/runtime/pull/72000 throw because we can't determine the Socket is connected at:

https://github.com/dotnet/runtime/blob/99297533df71e0d9e38ccf4a6bb6afd236a5f665/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs#L177-L178

I'm not sure what happens in the AcceptGetsCanceledByDispose case (may be similar). A stacktrace would help.

@antonfirsov can you take a look? I'm still on leave for a week.

[directhex]

There are plenty of indications that UNIX Domain Sockets are not actually supported on tvOS/iOS, and only work in the simulator by accident. The PRs you mention are part of the investigation in that regard. Waiting on some test results from Alex to determine whether the whole of UDS should be PNSE on one or both of those OSes.

[akoeplinger]

@directhex this is about normal sockets not Unix Domain Sockets.

I think @tmds's investigation makes sense and this is due to new cases added in https://github.com/dotnet/runtime/pull/72000. @antonfirsov @wfurt would you mind taking a look?

[wfurt]

I think we should disable the tests on affected platforms. I don't know why the UDS would not be connected. The second link make sense only when Socket is created from handle and I don't think this is the case.

[directhex]

OK, I've caught up with what's going on now. I'll ActiveIssue those with this bug.

[karelz]

@directhex thanks! Looking forward for the tests to be disabled. Please also triage this issue into appropriate milestone. Thanks!

[simonrozsival]

The tests were disabled in #73374

[jeffschwMSFT]

With the test being disabled, I am removing the blocking-clean-ci label, thanks