`System.Threading.Tasks.Dataflow.Tests` failing with NRE in `S.T.Tasks.ConcurrentExclusiveSchedulerPair.ProcessConcurrentTasks`

[radical]

trafficstars

Failures per Runfo - last 30 days:

Day	Run	Details
8/4	Rolling run AM - extra-platforms	Console log - Debian.10.Arm32.Open
8/4	PR 1924659	Console log - Alpine.314.Arm32.Open
8/2	PR 1919458	Alpine.314.Arm32.Open
8/1	PR 1916028	Alpine.314.Arm32.Open
7/29	Rolling run 1912276	Alpine.314.Arm32.Open
7/28	PR 1909906	Alpine.314.Arm32.Open

Potentially related: #73247 also begins failing on arm32 on 7/28. This issue first failed at 12:18pm, and #73247 starts ~5 hours later at 5:25pm. Both tests continue to fail ~once a day from this point onwards.

Hit on an unrelated PR - https://github.com/dotnet/runtime/pull/73423 . Build(Libraries Test Run checked coreclr Linux_musl arm Release), and log:

===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Threading.Tasks.Dataflow.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Threading.Tasks.Dataflow.Tests (found 328 of 336 test cases)
  Starting:    System.Threading.Tasks.Dataflow.Tests (parallel test collections = on, max threads = 4)
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Threading.Tasks.ConcurrentExclusiveSchedulerPair.ProcessConcurrentTasks() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ConcurrentExclusiveSchedulerPair.cs:line 461
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 938
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 77
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105
./RunTests.sh: line 168:    21 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Threading.Tasks.Dataflow.Tests.runtimeconfig.json --depsfile System.Threading.Tasks.Dataflow.Tests.deps.json xunit.console.dll System.Threading.Tasks.Dataflow.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Fri Aug 5 03:50:48 UTC 2022 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.

I could not find an existing issue for this.

Build	Kind	Start Time
1912276	Rolling	2022-29-07

[msftbot[bot]]

Tagging subscribers to this area: @dotnet/area-system-threading-tasks See info in area-owners.md if you want to be subscribed.

Issue Details

---

Hit on an unrelated PR - https://github.com/dotnet/runtime/pull/73423 . Build(Libraries Test Run checked coreclr Linux_musl arm Release), and log:

===========================================================================================================
/root/helix/work/workitem/e /root/helix/work/workitem/e
  Discovering: System.Threading.Tasks.Dataflow.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Threading.Tasks.Dataflow.Tests (found 328 of 336 test cases)
  Starting:    System.Threading.Tasks.Dataflow.Tests (parallel test collections = on, max threads = 4)
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Threading.Tasks.ConcurrentExclusiveSchedulerPair.ProcessConcurrentTasks() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/Tasks/ConcurrentExclusiveSchedulerPair.cs:line 461
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 938
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 77
   at System.Threading.Thread.StartCallback() in /_/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs:line 105
./RunTests.sh: line 168:    21 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Threading.Tasks.Dataflow.Tests.runtimeconfig.json --depsfile System.Threading.Tasks.Dataflow.Tests.deps.json xunit.console.dll System.Threading.Tasks.Dataflow.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem/e
----- end Fri Aug 5 03:50:48 UTC 2022 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.

I could not find an existing issue for this.

Author:	radical
Assignees:	-
Labels:	area-System.Threading.Tasks, blocking-clean-ci
Milestone:	-

[joperezr]

I'm taking a look at this by trying to analyze the dump so I can get which test is causing this.

[karelz]

@joperezr FYI: I updated top post -- it didn't seem to happen 8/13-8/24 at all.

[jkotas]

Likely fixed as part of https://github.com/dotnet/runtime/issues/73405#issuecomment-1206151902